Airflow in Action: Red Hat's Blueprint for Trusted AI Agents with Astro

At the Airflow Summit, Shoubhik Bose from Red Hat detailed how the company updated its data and AI platform using Apache Airflow on the Astro orchestration platform to streamline its complex internal infrastructure. Red Hat, traditionally a software company, faced challenges due to outdated data systems that could not keep up with its rapid growth and hybrid cloud integration. To address these issues, Red Hat developed the Dataverse platform, treating it like an open-source project with a data mesh architecture centered around tools like Snowflake, dbt, and Fivetran, while using Astro for orchestration. This approach allowed Red Hat to deliver real-time, reliable data to executives and support AI agents, such as a business analytics agent that enables natural language queries and a Privacy Impact Assessment agent for regulated industries. The platform's success relies on its integration with Airflow, dbt, and Astro Observe, which ensures data transformation reliability, orchestration across different vendors, and observability for non-technical stakeholders. By leveraging Airflow's capabilities, Red Hat has created a scalable and compliant data and AI infrastructure, enabling efficient data handling and paving the way for future AI developments.