March 2019 Summaries

Architecting Airflow infrastructure involves deciding between a monolithic or a multi-tenant setup, a choice driven by the limitations of Airflow's scheduler, which cannot be horizontally scaled unlike other components. Initially, Astronomer implemented a single massive Airflow deployment for managing various workflows, which proved unsustainable as the scheduler became a bottleneck due to its CPU-intensive nature and inability to handle silent failures effectively. Transitioning to multiple Airflow instances tailored to specific job requirements and using Kubernetes for container orchestration enhanced fault tolerance and availability, enabling quicker iterations and resource allocation adjustments. This polycentric design strategy, demonstrated by Metarouter's deployment of three separate Airflow environments, facilitates smoother operations and more efficient management of production and reporting tasks, although it acknowledges that there is no universal solution for all data infrastructure needs.

Astronomer's v0.8.0 release introduces significant updates to its platform, including a completely rewritten backend API designed for efficiency and future integrations, alongside an Elasticsearch, Fluentd, Kibana (EFK) stack for enhanced log management within the Astronomer UI. Currently, v0.8.0 is exclusive to Enterprise customers, as efforts to migrate the Cloud cluster to the new API are ongoing. Among the new features are faster deployment times due to the replacement of StatefulSets with Kubernetes Deployments, a Single Namespace Mode that optimizes Kubernetes resource usage, and support for Airflow 1.10.2. These updates are part of Astronomer's ongoing commitment to regular releases, with future enhancements and features continually being developed, as indicated on their Roadmap.