August 2024 Summaries

Task Runners, a significant enhancement to Kestra's orchestration capabilities, are now generally available, providing a solution for executing resource-intensive workflows using dynamically-provisioned resources in the cloud. These Task Runners allow users to offload computationally demanding tasks to on-demand compute services like AWS ECS Fargate, Azure Batch, and Kubernetes, thereby ensuring adequate resources while optimizing compute costs. This extensible ecosystem of plugins enables tasks to be executed in various remote environments with fine-grained resource allocation and a consistent API, facilitating seamless transitions from development to production. The development of Task Runners has benefited from extensive user feedback during its beta phase, leading to improvements in performance, usability, and integration with other Kestra features. With built-in support for multiple cloud providers, autocompletion, and documentation, Task Runners simplify workflow creation and offer flexibility for infrastructure evolution as needs change.

Kestra 0.18.0 introduces significant enhancements to both its Open-Source and Enterprise Editions, notably adding a Key-Value Store (KV Store) for data persistence across workflows, improving Namespace Management, and introducing new Outputs UI for better inspection of workflow artifacts. The Enterprise Edition gains new features such as SCIM Directory Sync for user and group synchronization, SQL Server backend support in preview, and enhanced audit logs with diff-based change tracking. Both editions benefit from improvements in Task Runners, now out of Beta, offering flexible task execution in various remote environments. The release also includes refined secrets management, extended cron scheduling with second-level precision, and numerous UI improvements aimed at enhancing user experience. New plugins for integrations with platforms like MinIO, GitHub, Jira, and Apache Airflow further expand Kestra's capabilities, supporting users in managing complex data workflows efficiently.