September 2026 Summaries

Kestra has launched a rebuilt Academy platform featuring refreshed Kestra Fundamentals content for version 2.0 and three new courses focused on infrastructure automation, AI workflows, and data pipelines. The courses are designed to help users build and operate workflows for specific use cases, with Fundamentals recommended for newcomers and specialized courses available for existing learners. The new platform offers improved lesson quality, code blocks, and navigation, while prior users will receive invitations automatically and retain completed Fundamentals progress. Each course concludes with a certification exam that awards a shareable Kestra certification upon passing.

A frontier engineer is presented as a role focused on designing and operating end-to-end business processes in which AI agents perform suitable tasks while humans and deterministic systems retain control over consequential actions. Unlike ML, software, platform, or prompt engineers, this role defines the boundaries between agentic and fixed steps, structures agent outputs for safe handoffs, applies approvals and permissions, monitors executions, and manages workflow changes through version control and deployment processes. Using incident response as an example, the text argues that agents can investigate logs and propose remediation, but production changes should be governed by validated inputs, human approval, controlled tools, rollback mechanisms, and complete audit records. It characterizes this work primarily as orchestration, distinguishing internal agent reasoning from the broader coordination of people, systems, policies, retries, observability, and governance. The article promotes Kestra as a workflow orchestrator for implementing these patterns, describing features for AI-agent tasks, approval gates, permissioned tools, monitoring, CI/CD, and reusable workflows, while concluding that organizations with multiple production agents likely already need someone to own these responsibilities regardless of the job title.

Kestra 2.0 expands its cloud plugin catalog beyond its long-standing AWS, Google Cloud, and Azure integrations, adding Cloudflare, Huawei Cloud, Clever Cloud, and DigitalOcean in response to a more international user base and growing data sovereignty requirements. Cloud plugins can support service tasks and triggers, enterprise task runners, internal object storage, and enterprise secret management, with the three major providers remaining the only integrations that cover all four layers. Google Cloud, AWS, and Azure offer container- and virtual-machine-based execution options, while Google’s Cloud Run runner includes a GCS logging alternative to avoid Cloud Logging API quotas. Huawei Cloud is the most comprehensive new provider, designed largely as an AWS-style counterpart with storage, data, messaging, compute, and infrastructure services; DigitalOcean focuses on provisioning and lifecycle operations; Cloudflare supports edge operations including Workers and AI inference; and Clever Cloud provides French PaaS deployment, logging, application, and operational triggers. Kestra also plans potential integrations for Scaleway, OVHcloud, and Oracle Cloud Infrastructure, while inviting users to request additional providers through its community channels.

Kestra 2.0 introduces a broad security overhaul alongside its rebuilt execution engine, driven by a systematic review that produced roughly sixty fixes and multiple published advisories. Its most significant architectural change removes database access from workers, which now communicate outbound with controllers over gRPC, limiting the impact of compromised workers and keeping secrets encrypted until use. The release strengthens authentication through bcrypt password hashing, constant-time credential comparisons, rate limiting, tighter access controls, and improved secret encryption and masking, while also addressing information leaks, stack-trace exposure, insecure file downloads, and previously open management endpoints. New protections include CSRF defenses for browser sessions, sanitization for Markdown, SVGs, and UI content to prevent XSS, configurable security headers, HTTP allow and deny lists to reduce SSRF risks, and safeguards against path traversal, ZIP bombs, ReDoS, SQL injection, and insufficient request validation. Although many protections are enabled by default, production deployments should configure initial authentication, worker TLS or mutual TLS, HTTP network restrictions, ZIP-bomb limits, and secure management-port access; Enterprise users can additionally restrict plugins and process execution in untrusted environments.

Kestra 2.0 expands its plugin system beyond task integrations, surpassing 2,000 plugins and introducing extensible policy rules, log data stores, file preview renderers, queue implementations, and UI artifacts that let plugins provide rich in-platform visualizations. Enterprise policy rules replace flow-level pluginDefaults, requiring users to migrate affected flows before upgrading, while configurable log stores can move execution logs out of the main database and support optional historical migration. Plugin authors can now create custom previews for formats such as Parquet or Avro, supply Vue-based UI components through the Artifact SDK, and organize large task forms with property groups, while the new slim container image reduces the default installation size and can automatically fetch plugins during local open-source use. Kestra maintains intentional backward compatibility with most 1.x plugins despite core changes such as Java 25, although several core execution-management tasks have moved to API-based replacements in the Kestra plugin SDK. The release also adds numerous cloud, AI, infrastructure, security, storage, and task-runner integrations, reflecting Kestra’s broader focus on orchestrating private infrastructure and operational workflows as well as data pipelines.

Kestra 2.0 introduces Cases, an Enterprise and Cloud feature that turns workflow executions, alerts, approvals, and remediation actions into centralized, auditable incident records within the orchestrator. Cases group repeated failures into a single incident with linked executions, assigned owners, watchers, acknowledgement and resolution SLAs, notification controls, status timelines, and required resolution reasons, while continued failures remain detected even after acknowledgement. They can be created from failed executions, successful-but-suspicious outcomes such as data-quality variances, external webhooks, approval workflows, and planned maintenance windows, with deduplication and auto-linking options to prevent alert floods. Operators can run predefined remediation flows directly from a case, preserving execution links and RBAC controls, while approvals can be tracked alongside HumanTask pauses. Cases uses Kestra’s existing primitives and permissions model, offers in-app notifications, and is positioned as an incident record rather than a full paging or ticketing replacement, as it currently lacks on-call rotations, escalation policies, and external ticket synchronization.

Kestra 2.0 is presented as an open-source, YAML-first workflow orchestration platform designed to run existing scripts and tools without requiring teams to rewrite them as Python-based pipelines. Its central architectural change is a Controller that allows workers to communicate through a single outbound, optionally mutual-TLS-secured gRPC connection rather than directly accessing the database, enabling deployment in separate clouds, on-premises systems, and restricted networks with no inbound connectivity. The platform supports nearly 2,000 plugins and tasks using languages such as Python, Go, Rust, SQL, and shell, while separating queue, repository, and log storage choices to improve flexibility and reduce database and queue traffic. Compared with Airflow and self-hosted Dagster, Kestra emphasizes its open-source outbound-only worker model, although Airflow retains a larger ecosystem and Dagster offers stronger Python-centric asset lineage and testing capabilities. The release also introduces AI Copilot for natural-language flow editing, AI Agent tasks, tools for external coding agents, and MCP resources for live documentation access. The open-source edition includes JDBC-based Postgres or MySQL support and the plugin catalog, while enterprise features include alternative queue backends, SSO, RBAC, multi-tenancy, worker groups, audit logs, and additional log-storage destinations.

Kestra 2.0, launched September 8, introduces a redesigned orchestration platform intended to unify data, infrastructure, application, and business-process workflows under one control plane, addressing fragmented ownership, audit trails, and governance across specialized tools. The release replaces Kestra 1.x’s separate JDBC and Kafka Streams engines with a single architecture that separates queues from repositories and adds a Controller, allowing workers to connect outbound through mutual TLS without direct database access and to run across clouds, on-premises systems, restricted networks, or air-gapped environments. It adds integrated case management for incidents, approvals, SLAs, remediation actions, and deployment promotion, creating audit trails that connect automated executions with human decisions. Kestra 2.0 also positions AI agents as governed tasks within workflows, supporting approval gates, MCP integrations, external tool calls, natural-language interaction with flows and logs, and publication of workflows as typed agent-callable tools. Existing 1.x users can retain their flows while using automated server migration, a YAML flow-rewriter CLI, migration documentation, and agent-assisted support for more complex changes; core engine, UI, editor, and plugins remain Apache 2.0, while Enterprise provides additional governance, backends, and multi-tenancy.

Kestra 2.0 is an LTS release that redesigns the orchestration platform around a separated control plane and data plane, enabling workers to connect through outbound gRPC streams without direct database access and allowing deployment across regions, clouds, and restricted networks. It decouples queues from repositories, supports multiple backend combinations, and simplifies the underlying execution architecture while introducing security options including TLS, mTLS, and JWT authentication. Major additions include MCP-based AI tool triggers, a multi-mode AI Copilot, an improved no-code editor, real-time execution monitoring, action-level RBAC, namespace-scoped governance policies, integrated incident Cases, UI-based flow promotion with drift detection, Git version control for blueprints, expanded IAM CLI commands, tag-based Worker Groups, new cloud and VM task runners, and plugin-provided UI artifacts. Flow authoring changes replace ForEach tasks with isolated Loop sub-executions and trigger condition chains with Pebble when expressions, while storage cleanup, external log stores, slim container images with plugin auto-installation, reusable inputs, quotas, locking, enhanced testing, and observability features broaden operational capabilities. Upgrades require Kestra 1.3.x, with migration tooling handling many mechanical changes, though removed constructs such as ForEach, pluginDefaults, trigger conditions, and workerGroup.key require review or conversion; the release includes migration guides and receives bug and security backports for one year.

Kestra 2.0 rebuilds its execution engine to reduce the work required per execution, delivering substantially higher performance than version 1.3 on identical hardware and database configurations. In benchmarked simple flows, sustained throughput doubled from 2,000 to 4,000 executions per minute while standalone latency fell from roughly 150 ms to 120 ms and p99 latency at 2,000 executions per minute dropped from 694 ms to 159 ms; complex flows improved throughput by 50 percent and latency by about 25 percent, while a 100-iteration Loop ran three times faster. These gains stem from a simplified single-consumer queue model, lightweight command-based messages rather than serialized executions, externalized task outputs, more efficient Executor database operations and caching, larger default connection pools, and a gRPC-based worker architecture that prefetches and batches jobs, logs, and metrics. Although PostgreSQL row-lock contention remains the primary scalability limit when it serves as both repository and queue, separating the queue through RabbitMQ or Redis maintains low, stable latency through 5,000 executions per minute with moderate CPU use, shifting the recommended path for workloads beyond 4,000 executions per minute from larger Kestra nodes to dedicated queue backends.

Kestra 2.0 focuses on connecting existing AI coding agents, such as Claude Code, Cursor, and Codex, to governed workflow automation rather than providing a proprietary agent. Its open-source MCP Tool Trigger can expose a Kestra flow as an MCP tool, automatically mapping flow inputs and outputs while recording every agent invocation as a standard execution with logs, retries, RBAC enforcement, audit labels, and optional human approval gates through Cases. Kestra provides a default managed MCP server and, in Enterprise, configurable additional servers with authentication, access controls, system instructions, and selectable exposed flows. A separate public MCP server and Agent Skills package help coding agents access live Kestra schemas, documentation, blueprints, flow-generation safeguards, operational commands, and Airflow migration support. The release also expands configurable AI provider support, including self-hosted and network-isolated models, adds a cautious multi-mode Copilot interface with explicit confirmation for changes, and enhances in-workflow AI Agent tasks with deterministic guardrails, Prometheus metrics, and external MCP client capabilities. Open-source features include MCP workflow exposure, agent resources and skills, AI Agent tasks, and limited Copilot options, while Enterprise adds broader provider support, advanced security controls, multiple MCP servers, Copilot RBAC, and approval Cases.

Kestra 2.0 decouples its queue and repository backends, replacing the previous bundled JDBC or Kafka-plus-Elasticsearch choices and their separate engine implementations with a single executor, scheduler, and indexer that provide consistent behavior across configurations. The recommended default remains Postgres, which serves both queue and repository functions for most deployments, while AMQP/RabbitMQ or Redis paired with Postgres can reduce queue latency, and Kafka paired preferably with Postgres supports higher throughput and horizontal scaling without requiring Elasticsearch. Elasticsearch or OpenSearch remains an option for repository workloads that benefit from search and retention capabilities, although Postgres offers lower execution-state latency through immediate read-after-write consistency. Queue changes include a single consumer group per queue, default failure of messages exceeding 1 MB, and the ability to skip unprocessable messages. Open-source users must use one JDBC backend for queues, repositories, and logs, whereas Enterprise users can independently select backends and external log stores; future support is planned for separate instances dedicated to queue and repository workloads. Backend switches do not require flow changes but require draining or handling in-flight executions, and workers now connect outbound to the controller without database access or inbound worker ports, enabling deployment across segmented or remote environments.

Kestra reviewed seven plugins to reduce repeated hot-path work and address silent data-loss or error-reporting failures, using JMH benchmarks and real local services to measure performance changes where applicable. JDBC now pools remote database connections by default and caches result-set metadata per result set, yielding roughly 20–25 times faster connection operations with about 200 times less allocation and improved row-mapping throughput. Serdes reuses date formatters and an error-path JSON mapper, reducing parsing overhead and making exception-message processing about 66 times faster, while Redis batches list pushes into variadic LPUSH commands, reducing 10,000 pushes from 10,000 round trips to 20 and substantially improving latency-sensitive workloads. MongoDB loading now passes maps directly as documents instead of serializing and reparsing JSON, producing roughly a 25-fold speed improvement and far less garbage. Reliability fixes ensure GCP Pub/Sub waits for in-flight messages during shutdown, BigQuery supports concurrent appends, AWS Athena and S3 paginate beyond 1,000 results, SQS batches sends, and Kafka propagates producer failures rather than reporting successful tasks when messages were not delivered.

Kestra 2.0 replaces the platform’s prior engine architecture to address database-dependent workers, queue bottlenecks, and the maintenance burden of separate JDBC and Kafka Streams implementations. The new design decouples queue and repository backends, retaining JDBC support for open-source deployments while offering Redis, AMQP, and Kafka queue options in Enterprise, with a single executor, scheduler, and worker implementation across configurations. A new Controller service intermediates communication between workers and backend systems through secured outbound gRPC connections, allowing workers to operate in separate clouds, regions, on-premises environments, or restricted networks without direct database access or exposed inbound ports. The release also reduces queue traffic by loading task outputs only when required, introduces an Indexer Service for non-JDBC backend configurations, supports workload routing through Worker Groups, and enables logs to be stored separately in JDBC, Elasticsearch, Datadog, or Splunk to reduce pressure on primary databases.