December 2018 Summaries

The Astronomer team has adopted Apache Airflow as a critical tool for managing data workflows, particularly for their product Metarouter.io, an event-routing platform. Initially, they faced challenges in handling batch-processing needs for data pushed from web apps to Amazon Redshift, which led them to explore various solutions including Airflow. This open-source tool offered features such as dynamic task generation, scalability, dependency management, and robust error handling, making it ideal for Metarouter’s requirements. The team detailed their use of Airflow to route events from apps to Redshift, involving steps like transforming data via Vortex, storing it in S3, and loading it into Redshift using dynamically generated Directed Acyclic Graphs (DAGs). They further optimized their system by switching from DC/OS to Kubernetes, improving resource usage with the Celery Executor, and enhancing monitoring capabilities using Prometheus. Despite Airflow’s quirks, the team remains committed to refining their platform and contributing to the Airflow community, focusing on real-time log streaming and efficient resource usage through Kubernetes.

The release of Astronomer v0.7.0 introduces several new features and enhancements aimed at improving user experience and functionality. Key highlights include the ability for users to select their executor and adjust resource allocations directly from the app's user interface, as well as initial support for Airflow 1.10.1. The latest version also brings new Grafana dashboards for Enterprise Edition users, enabling better monitoring of persistent storage, Prometheus, registry, Fluentd, and Airflow container states. Additionally, the update supports alerts based on Prometheus data for platform and Airflow deployments, allows injecting Airflow environment variables at runtime, and includes functionality for Airflow chart upgrades. The development team emphasizes its commitment to regular updates and encourages users to stay informed through their Roadmap.