March 2018 Summaries

The Astronomer Platform is a managed service for Apache Airflow® designed to address the evolving needs of data teams dealing with increasingly large and fast data sets. As traditional ETL tools become insufficient for flexible data processing, Airbnb data engineer Maxime Beauchemin developed Airflow to handle workflow management through code, offering a more customizable solution. However, Airflow's implementation can be challenging, prompting Astronomer to create a Managed Airflow product that focuses on secure deployment, scaling, providing a top-notch developer experience, and enabling data teams to work across diverse data environments, whether in the cloud or on-premises. This platform is aimed at enhancing the ease and efficiency of running data pipelines without the burden of infrastructure management.

Astronomer has introduced the Astronomer Enterprise Edition, a comprehensive data engineering platform designed to enable software engineers to efficiently create and manage data pipelines without concerns about infrastructure security and stability. The platform features a Heroku-like CLI for pipeline deployment and a New Relic-like interface for monitoring and troubleshooting, and it is built on open-source technologies like Kubernetes, Airflow, Prometheus, and Grafana. It also offers a library of open-source Airflow connectors developed over the past year, emphasizing collaboration, version control, and community engagement, akin to modern software projects. Users can easily deploy the platform following provided documentation or explore Astronomer Open, a local version available via Docker Compose. In addition to current features, the company plans to integrate a Clickstream module into the Enterprise Edition soon, inviting interested parties to visit their enterprise page or contact them for more information on Airflow and data engineering.