March 2024 Summaries

The Astronomer Champions Program for Apache Airflow has welcomed its second cohort of hand-selected individuals who embody the spirit of championing the Apache Airflow project. These 23 new members bring a diverse global presence and extensive expertise in Apache Airflow, having worked on various aspects such as data engineering, cloud computing, and software development. They have been selected for their passion for sharing insights and contributing to the growth and innovation of the community through blogging, presenting, hosting meetups, and actively engaging with other users. The new cohort includes experts from various industries and organizations, including Disney Streaming, Petco, CRED, DataRock Labs, and Astronomer, among others. They will work closely with Cohort 1 to further enhance Airflow's capabilities and promote its adoption in the data engineering community.

The latest Astro Platform release introduces several key enhancements aimed at bolstering governance, fortifying security, and accelerating innovation. The release includes new features such as Organizational Dashboards for comprehensive insights into platform usage, the Astro API for secure automation of environment management and integration at scale, custom deployment roles for granular access control aligned to security requirements, customer managed workload identity for secure authorization to data services from Airflow on the platform, and a GitHub integration to streamline CI/CD processes for developers. These features are designed to support Enterprise organizations that require additional features and tighter support for SLAs, with some features only available on the Enterprise product tier.

SnowPatrol is a Snowflake anomaly detection and cost management application powered by Machine Learning and Airflow. It aims to help users proactively identify abnormal usage and simplify root-cause analysis and remediation. The solution uses an Isolation Forest model to detect anomalies in Snowflake usage, with data exploration, feature engineering, model training, and prediction managed through distinct Airflow workflows. Data-aware scheduling and dynamic task mapping are used to optimize resource utilization and adapt to changing requirements. The application also leverages Weights and Biases for experiment and model tracking, and is designed to be scalable, flexible, and customizable. By automating anomaly detection and alerting, SnowPatrol helps users avoid overages, reduce their Snowflake costs, and improve their data engineering practices.

Ask Astro, a RAG-based chat assistant leveraging large language models, has been a popular tool for inquiries related to Astronomer products and Apache Airflow, but as its usage grew, challenges related to document retrieval and answer accuracy emerged. Initially, the system utilized a combination of LangChain, GPT-3.5, GPT-4, and OpenAI's text-embedding-ada-002 for document retrieval, but it occasionally faced issues with irrelevant document retrieval, particularly when certain queries didn't align well with the embedding model. To address these challenges, a hybrid search approach was introduced, combining BM25 sparse vector search with dense vector search, which improved retrieval accuracy by leveraging both keyword-based and vector search techniques. This hybrid search allowed for a larger pool of candidate documents, enhancing the robustness of the system and enabling more accurate re-ranking of documents using Cohere Rerank. With Cohere Rerank, the system processes user prompts and document content to calculate relevancy scores, ultimately retaining the top eight highest-scoring documents, significantly improving the accuracy and relevance of the responses. The integration of hybrid search and Cohere Rerank resulted in a 13.5% improvement in response accuracy and effectively addressed previous issues, setting a new standard for Ask Astro's answer accuracy and document relevance.