June 2024 Summaries

Kestra's introduction of Realtime Triggers in version 0.17.0 enhances its capability to instantly respond to events without the need for polling, making it particularly beneficial for business-critical workflows that demand millisecond latency reactions. This feature supports various queuing systems, including Apache Kafka, RabbitMQ, and cloud-based services like AWS SQS, allowing seamless orchestration of processes such as fraud detection and order processing. The blog demonstrates a practical application in an e-commerce scenario where Realtime Triggers are used to enrich incoming order events with product details from Cassandra and insert them into MongoDB. The process leverages Kafka for event streaming and includes a Kestra flow with tasks to fetch product data and create detailed order records, showcasing the efficiency of Realtime Triggers in achieving real-time data processing and orchestration.

Kestra has introduced Realtime Triggers, marking a significant advancement in orchestration by enabling real-time workflow execution with millisecond-latency integrations across various messaging systems and SQL databases. This feature addresses the limitations of traditional data orchestration solutions, which often struggle with the demands of real-time processing and rely on inefficient sensors, by offering seamless integration without the need for custom sensors. Realtime Triggers allow businesses to respond instantaneously to events, enhancing applications like fraud detection and real-time recommendations, while providing full observability for monitoring and troubleshooting. Kestra simplifies the setup process with API-first configurations, enabling users to quickly deploy real-time workflows and maintain operational efficiency. The new capabilities also include Change Data Capture with Debezium, support for the outbox pattern to ensure reliable message delivery, and coordination of microservice interactions, thereby improving responsiveness and integration across diverse systems. By unifying batch and real-time processing, Kestra allows organizations to transition smoothly between these modes, reducing latency and simplifying workflow management for both developers and business users.

Quadis, Spain's largest car retailer, has significantly enhanced its operations by transitioning to the orchestration platform Kestra, replacing an older in-house solution. Kestra's capabilities have enabled Quadis to automate critical processes such as financial reporting and customer order notifications, ensuring seamless integration and reliable task execution across various departments. This shift has allowed Quadis to adopt best software practices like code versioning and separation of business logic from orchestration, while also accommodating users with varying technical expertise through its UI and Git integration. Over a short period, Quadis onboarded multiple developers to Kestra, leveraging its integration with Docker and AWS to streamline operations and plan for future expansions beyond Spain. The transition is part of a broader modernization effort that includes establishing a new data platform, Atlas, to support advanced analytics and resource optimization, positioning Quadis for further growth in Europe.

Kestra's latest version control integration introduces enhanced Git tasks designed to elevate GitOps capabilities within workflows by simplifying processes and reducing human error. The new features, such as PushFlows and SyncFlows, allow seamless synchronization of development and production environments, ensuring that code is consistently updated and stored securely in Git repositories. Users can leverage dry-run mode to validate workflows before committing changes, thereby increasing precision and reducing mistakes. The integration supports multiple patterns, offering flexibility for those utilizing Git as a single source of truth, whether through Git Sync for automated updates or Git Push for direct UI editing, and seamlessly fits into existing CI/CD pipelines with tools like GitHub Actions or Terraform. Comprehensive guides and video tutorials are available to assist users in adopting these new Git tasks, enhancing productivity and collaboration in managing automation pipelines.

Kestra 0.17.0 introduces significant enhancements to the workflow orchestration tool, including a new, faster Code Editor that improves the user experience by offering unified development for Namespace Files and Flows, and autocompletion for templated expressions. The update features improved Git integration for more flexible workflow management, real-time event triggers for immediate reaction to business-critical events, and new orchestration capabilities with the WaitFor task, which repeatedly executes tasks until a specified condition is met. The Pause task now supports onResume inputs for human-in-the-loop processes, and the release also includes UI improvements with a guided onboarding experience and an enhanced Settings page. Kestra now runs on Java 21, includes an ARRAY input type for better data handling, and offers improved plugin functionalities. Additionally, Kestra 0.17.0 enhances execution management, supports new plugin tasks, and introduces deprecations for outdated features, encouraging users to adopt the new functionalities for optimized workflow orchestration.