October 2023 Summaries

The blog post discusses the integration of vector databases, specifically Weaviate, with Kestra, a workflow orchestration tool, highlighting the relevance of vector databases in modern AI applications due to their ability to efficiently store and query vector embeddings. It introduces the Kestra Weaviate plugin, which facilitates interaction with the Weaviate vector database, allowing users to create schemas, insert records, and perform queries within Kestra workflows. Vector databases, such as Weaviate, provide scalability and performance necessary for handling AI tasks like image recognition, knowledge base creation, and recommendation systems. The article provides a practical guide on implementing these operations using Kestra, showcasing how to set up and manage data in a Weaviate vector database, and emphasizes the growing importance of understanding vector databases for IT professionals involved in AI and machine learning projects.

In 2023, companies have a plethora of orchestration tools at their disposal to manage the increasing complexity of data workflows from various sources, which is crucial for making data-driven business decisions. The article explores five major orchestration tools: Apache Airflow, Kestra, Azure Data Factory, Prefect, and AWS Step Functions, highlighting their unique features, suitability, advantages, and drawbacks. Apache Airflow offers a robust open-source solution with a strong community, though it requires familiarity with Python. Kestra provides a language-agnostic platform using YAML, making it accessible for non-technical users, while Azure Data Factory and AWS Step Functions excel in cloud-based operations with seamless integrations into their respective ecosystems. Prefect, another Python-centric tool, is praised for its flexibility and dynamic workflows. Each tool has its specific strengths, such as scalability, integration capabilities, and ease of use, but they also come with certain limitations like cost and platform dependency, underscoring the importance of aligning the choice of tool with the specific needs and existing infrastructure of an organization.

Over the past decade, job scheduling has evolved from basic cron jobs to sophisticated platforms that cater to modern engineering needs, such as microservices and cloud-native technologies. The text examines three scheduling platforms—Dollar Universe, Control-M, and Kestra—highlighting their capabilities and limitations. Dollar Universe, by Broadcom, excels in enterprise-scale scheduling across various platforms but struggles with cloud-native architectures and dynamic scaling. Control-M, from BMC Software, offers robust workload automation with features like file transfer management and batch processing but bears legacy overhead that hampers agility in complex workflows. Kestra, a modern solution, stands out for its flexibility and Infrastructure as Code practices, allowing for seamless scheduling, monitoring, and management of workflows with support for complex conditions and backfill capabilities. The choice of scheduling platform depends on various factors, including task complexity and system architecture, with Kestra providing a more adaptable option for contemporary requirements.

A Major League Soccer (MLS) club has effectively leveraged Kestra, an automated data orchestration tool, to enhance its football data analytics capabilities, driving both athletic and financial decisions. By transitioning from a challenging Papermill notebook-based system to Kestra, the club now efficiently manages data pipelines, processes various types of data, and utilizes machine learning models to support player recruitment, match analysis, and scouting. Kestra's web-based control plane enables seamless creation and monitoring of workflows, facilitating the ingestion of data into AWS S3 and a PostgreSQL server. Additionally, the tool allows for the easy management of bespoke KPIs and supports polyglot programming through Docker containers, which streamlines the integration of Python and R codebases. As the analytics team moves towards developing a possession value model to quantify player contributions, the club continues to optimize its performance and strategy, benefiting from Kestra's scalability and versatility.

Data ingestion, a critical yet challenging task for data engineers, involves numerous complexities such as evolving schemas, API changes, and unexpected data volume fluctuations, often requiring manual intervention without much recognition. Several tools like Fivetran, Airbyte, Singer, CloudQuery, and dltHub have emerged to mitigate these challenges by offering different approaches to data integration. While no-code tools like Fivetran simplify the process for business users, they often lack the customization needed by engineers, which is where open-source solutions like Airbyte, CloudQuery, and dltHub come into play by allowing more flexibility and community contributions. Despite the advancements, the problem remains inherently unsolved due to commercial dynamics and the evolving nature of APIs, though emerging AI-powered tools may offer more scalable solutions in the future. Kestra, an event-driven orchestrator, partners with these tools to streamline the orchestration of data ingestion pipelines, demonstrating the progress facilitated by open-source communities in tackling these persistent challenges.

The blog post discusses integrating SurrealDB, a NewSQL multi-model cloud database, with Kestra, a data orchestration platform, highlighting the versatility and features of SurrealDB, such as its support for tables, documents, and graphs without the need for traditional SQL JOINs, and its suitability for various applications including serverless, jamstack, and single-page apps. It details the installation process of SurrealDB, its use of the SurrealQL query language, and steps to create and query records. The article then demonstrates how to set up basic and advanced data flows in Kestra using the SurrealDB plugin, showing how to perform data retrieval and insertion with simple queries, set default plugin values, and utilize advanced database features like controlling record ID values, using string functions for derived attributes, and linking records across tables. The post concludes by emphasizing the growing library of Kestra plugins and the ease of integrating these technologies to streamline data pipeline creation without restricting users to a specific programming language.