Migrating from Airflow to Kestra with AI Coding Agents

Airflow 2 has reached its end of life, prompting many teams to consider upgrading to Airflow 3, although this transition poses challenges due to the complexity of migration and the opportunity to evaluate alternative solutions like Kestra. The author suggests that the traditional process of upgrading often fails because teams are hesitant to deviate from the status quo without a working example to justify the shift. However, the introduction of large language models (LLMs) and tools like migration-skills have streamlined this process, allowing for a quicker conversion of Airflow DAGs to Kestra flows, thus facilitating easier demonstration and evaluation of Kestra's benefits. Kestra offers a different approach with declarative YAML flows, file-based data passing without size limits, and a rich plugin ecosystem that enhances integration capabilities, supported by an IDE-like UI that improves workflow management. The migration process involves using AI coding agents to convert Python DAGs into Kestra YAML flows, which are validated against live schema to ensure accuracy, and deploying the resulting configurations using the Kestra CLI, allowing for a structured transition from Airflow to Kestra with improved data handling and execution visibility.