January 2024 Summaries

In 2024, data engineering is experiencing significant shifts driven by AI integration, economic pressures, and evolving job roles, prompting data teams to adopt more versatile skills and cloud-native technologies. These trends are leading to a convergence of software and data engineering roles, with increased emphasis on efficiency and the need for data teams to transition from cost centers to profit centers by creating AI-enhanced, revenue-generating products. AI is reshaping the job market, with companies reallocating resources towards AI development and fostering growth in hybrid roles that blend traditional data engineering with software and platform engineering. The adoption of event-driven systems and data lakehouses is rising due to budget constraints and the demand for real-time data processing, while GitOps is becoming a popular methodology for managing data infrastructure changes. The competitive landscape in data tools is pushing companies to balance open-source and commercial offerings, with open-source LLMs gaining traction alongside more affordable commercial AI applications.

Valorissimo, a prominent platform in the real estate sector, has significantly enhanced its marketing and sales operations by developing a modern data stack in collaboration with Hanalytics and Kestra. This innovative approach has doubled their conversion rates by integrating specialized tools like Airbyte for data ingestion, BigQuery as a data warehouse, dbt Core for data transformation, Hightouch for reverse ETL, and Looker Studio for visualization. Kestra plays a crucial role in orchestrating these processes, ensuring seamless data flow and quality through advanced workflow automation. This strategic partnership and technological integration enable Valorissimo to maintain its competitive edge in the real estate investment sector by providing personalized marketing automation and efficient data-driven insights.

Kestra 0.14.0 introduces significant updates, including enhancements to the embedded VS Code Editor and the integration of automated Git sync for improved version control, allowing users to easily sync and push code between Git and Kestra. The update also brings new functionalities such as built-in documentation, improved website search, and plugin homepages. Notably, recursive rendering of Pebble expressions is no longer default, giving users more control over expression rendering, while execution labels can now be set dynamically from tasks for better observability. The release also incorporates a new Allow Failure pattern as a core task property and enhanced scheduling conditions for executing workflows only on working days or public holidays. Additional features include support for the SMB protocol in the fs plugin, a new Apache Druid plugin, and enhancements to existing plugins like OpenAI and Google Vertex AI, which now support more complex workflows through function calling and multimodal completion. The update is marked by community contributions, reflecting the collaborative nature of the Kestra project.

Gorgias employs Infrastructure as Code (IaC) principles to enhance its data engineering practices using various tools like Kestra, Airbyte, dbt, Hightouch, and Terraform. By adopting a declarative approach, Kestra transforms data pipeline management, offering a seamless integration with Gorgias' existing toolkit, which includes BigQuery, PostgreSQL, and GitHub. This method simplifies task creation through an intuitive UI and supports native Webhook triggers, allowing modularity and versioning in workflows. Gorgias has shifted to a dedicated data orchestration tool to address the limitations of embedded schedulers, maintaining complex logic across systems with a centralized platform for monitoring and debugging. Terraform modules and Kestra subflows help abstract complexities for users, facilitating modular development and efficient configuration management. Kestra's integration as a CI/CD engine further supports Gorgias' infrastructure management, enabling streamlined operations and enhanced collaboration among data practitioners.

Integrating Sentry.io with Kestra workflows enhances observability by allowing users to monitor and alert on workflow executions through logs, metrics, and traces. Kestra offers two plugins, SentryAlert and SentryExecution, for sending issues to Sentry, helping troubleshoot failures and monitor successes. SentryAlert can be utilized for any reason, including workflow failures, by sending customizable JSON payloads to Sentry's API. SentryExecution is particularly useful when monitoring multiple workflows, as it can be triggered by any flow within a specified namespace that fails or generates warnings. This integration is made easy with Sentry's zero-dollar plan, allowing developers to register and start monitoring without initial costs. Together, Kestra and Sentry provide a comprehensive solution for enterprise-grade workflow automation and end-to-end observability, offering real-time monitoring and alerts that can be tailored to individual use cases.