The Astronomer team has adopted Apache Airflow as a critical tool for managing data workflows, particularly for their product Metarouter.io, an event-routing platform. Initially, they faced challenges in handling batch-processing needs for data pushed from web apps to Amazon Redshift, which led them to explore various solutions including Airflow. This open-source tool offered features such as dynamic task generation, scalability, dependency management, and robust error handling, making it ideal for Metarouter’s requirements. The team detailed their use of Airflow to route events from apps to Redshift, involving steps like transforming data via Vortex, storing it in S3, and loading it into Redshift using dynamically generated Directed Acyclic Graphs (DAGs). They further optimized their system by switching from DC/OS to Kubernetes, improving resource usage with the Celery Executor, and enhancing monitoring capabilities using Prometheus. Despite Airflow’s quirks, the team remains committed to refining their platform and contributing to the Airflow community, focusing on real-time log streaming and efficient resource usage through Kubernetes.