August 2018 Summaries

Astronomer has released version 0.4.1 of its platform, introducing key enhancements in authentication, stability, and monitoring. The update integrates Auth0, allowing users to authenticate via local, Google, or GitHub accounts, simplifying the login process. Improvements to the Airflow webserver include resolving stability issues and implementing a comprehensive health check to ensure consistent uptime. Additionally, the Grafana monitoring tool has been updated to version 5.2.2, featuring Elasticsearch alerting, an improved Docker image, and enhanced security measures. The platform now also supports increased volume capacity for Celery workers, enabling them to manage larger workloads without additional configuration. Users interested in upcoming features can refer to the Astronomer Roadmap guide or join the beta program to explore the platform further.

Apache Airflow, originally developed at Airbnb in 2014 and open-sourced in 2015, is a leading open-source workflow management system that enables users to author, schedule, and monitor data pipelines using Python. As it gears up for its anticipated 2.0 release, the project continues to emphasize configurability, extensibility, and scalability. Short-term improvements for version 1.10 include enhancements in role-based authentication controls (RBAC) and user interface updates, as well as the introduction of the Kubernetes Executor, which allows for auto-scaling of workers via Kubernetes. Looking further ahead, the project aims to establish first-class API support and make Airflow fully cloud-native, with efforts to incorporate databases like CockroachDB to ensure high availability and scalability. The community also seeks to improve its testing suite and entertain the idea of a plugin manager to better handle the growing ecosystem of plugins. These initiatives underscore Airflow's commitment to maintaining its flexibility and reliability as a scheduler across diverse deployment scenarios.