September 2023 Summaries

Kestra 0.12.0 introduces a range of new features and enhancements to improve modular, event-driven, and containerized workflows, including an updated subflow UX for easier navigation and integration with Docker and AWS EventBridge. The release adds new plugins such as Docker and SurrealDB, along with AWS integrations including Lambda, Amazon EventBridge, and Amazon Kinesis Data Streams, to facilitate the orchestration of microservices and real-time applications. It also includes worker task heartbeats for better handling of infrastructure downtime and improved dbt tasks with an option to disable parsing of the dbt DAG. The update enhances the UI, offering better display of logs, JSON columns, and warnings for deprecated properties, and introduces AWS, Azure, and Google Cloud Secrets Managers in the Enterprise Edition to securely store Kestra secrets. These advancements aim to enhance workflow visibility, reliability, and maintainability, catering to both open-source and enterprise users.

Kestra has announced its new partnership with Snowflake, a leading provider in cloud data warehousing, to enhance data workflow management through integrated solutions. This strategic alliance leverages Snowflake's robust platform for handling structured and semi-structured data and Kestra's expertise in workflow orchestration, providing a comprehensive solution that enhances efficiency and reliability in managing data pipelines. The collaboration introduces event-driven triggers that allow real-time data changes to initiate workflows, simplifying file management and facilitating SQL queries through Kestra’s Snowflake plugin. This integration supports both ETL and ELT operations, making data warehousing accessible even for non-developers by utilizing YAML. The partnership aims for long-term collaboration with plans for more advanced features and capabilities to optimize data management further, signaling a forward-looking vision that promises to continue delivering value to customers.

Kestra and HashiCorp's Terraform have formed a strategic partnership to enhance workflow automation through the Everything-as-Code paradigm, allowing for streamlined CI/CD processes for operational and analytical workflows. By integrating Kestra's environment management capabilities with Terraform's Infrastructure as Code approach, this collaboration enables seamless and atomic deployments, aligning operations with DevOps principles. The partnership not only simplifies the transition of data pipelines from development to production by addressing challenges such as environment setups and resource permissions but also promotes robust software delivery practices. As demonstrated by Leroy Merlin France, the integration has led to significant improvements in workflow automation, version control, and data activity, showcasing the potential of this collaboration to boost productivity and team confidence in production settings.

Kestra has launched a partner program to expand its influence in orchestrating applications and mission-critical operations, aiming to build strong partnerships with organizations that share its vision for workflow automation. This initiative seeks to provide extensive support to businesses dealing with implementation, re-platforming, migration, and custom development projects, with Converteo and NTICO joining as inaugural consulting partners. The program underscores Kestra's commitment to integrating with modern data stack tools like Airbyte, and emphasizes the solution's efficiency in data stream processing, scalability, and client support. By collaborating with technology and consulting companies, Kestra aims to enhance customer value through trust-based relationships and deliver comprehensive solutions across various industries.

Accessing and analyzing football data has become more accessible with tools like Kestra, Malloy, and DuckDB, which streamline data scraping, processing, and analysis. Fbref.com offers a straightforward scraping process due to its user-friendly design and scraper policies, allowing data to be extracted and formatted into CSV files. Malloy, a new open-source language, simplifies data analysis by handling complex queries and data hierarchies while retaining SQL-like features, making it easier to compose reusable and nested computations. The orchestration tool Kestra integrates these processes by scheduling data scraping and analysis workflows using YAML files, enabling automated and seamless data management. This combination exemplifies a modern data stack, where domain-specific languages like Malloy and declarative orchestrators like Kestra enhance user experience by allowing fluid, semantic expression of data needs.

Kestra is a declarative orchestration tool designed to improve the data experience by integrating software engineering best practices into data workflows. It addresses common challenges in the modern data ecosystem, such as resource allocation and pipeline latency, by treating data as a product and adopting frameworks like data mesh for scalable operations. Kestra enhances data observability and modularity, providing a robust layer of control to ensure a unified data landscape. It supports linear scheduling, event-driven flows, manual overrides, and universal integration, making data orchestration adaptable to evolving business and data models. Kestra emphasizes a superior developer experience, focusing on role-specific functionality, data trust, logic separation, and speed. By enabling a cohesive and efficient data choreography, Kestra aims to create an ecosystem where tools work in harmony while being individually effective, ultimately simplifying data management and making it more user-friendly and seamlessly integrated into business processes.

Ntico, a French consulting and software company, has developed Locxia, a data platform that utilizes Kestra to process and visualize extensive geospatial datasets from satellite imagery and public data. Locxia is designed to provide businesses and local authorities with comprehensive insights into territories and demographics, aiding in customer database enhancement and effective public policy communication. Kestra serves as the backbone of Locxia, enabling efficient data ingestion from diverse sources such as APIs and open-data platforms, and storing the data in an AWS S3 repository. Its user-friendly interface and customizable features allow Ntico to create tailored data pipelines, integrating seamlessly with GitHub for continuous development. The collaboration has resulted in a scalable, cost-effective solution that empowers users to handle vast data volumes and make data-driven decisions.

Kestra has announced a technology partnership with Airbyte, aiming to enhance the Modern Data Stack by ensuring seamless data transfer, task execution, and process automation. This collaboration leverages the strengths of both open-source platforms to simplify data pipeline creation, providing a unified platform for end-to-end data orchestration. The partnership introduces Kestra's Airbyte plugin, which allows users to trigger Airbyte syncs and integrate data ingestion processes into comprehensive workflows. Emphasizing an "Everything as Code" approach, the integration supports efficient data management and infrastructure deployment with robust orchestration capabilities, including parallel syncs, scheduled ingestion, and complex data processing tasks. Both Kestra and Airbyte offer strong Terraform support, enabling users to manage resources using declarative configuration files, thus extending the possibilities for data professionals and aligning with diverse organizational needs.

Reverse ETL and Customer Data Platforms (CDPs) play pivotal roles in modern data management by transforming raw data into actionable insights. Reverse ETL focuses on pushing curated data from warehouses back into operational systems, making it actionable for business users, whereas CDPs serve as centralized repositories that consolidate customer data for a unified view, aiding in marketing, sales, and service efforts. While both have overlapping functionalities, CDPs excel in providing a comprehensive understanding of customers, whereas Reverse ETL tools enable operational analytics by utilizing analytical data across various platforms. Notable platforms such as Segment, Tealium, BlueConic, and Hightouch exemplify the capabilities of these technologies, with Hightouch uniquely merging Reverse ETL and CDP features to offer advantages like real-time syncing and flexible data integration. As these platforms evolve, they are expected to blur the lines between their distinct functionalities, potentially incorporating more advanced features and customization options.