February 2024 Summaries

Dosu, an AI company, uses Astronomer to manage its data pipelines and integrate dbt with Airflow, allowing it to build advanced data workflows and provide accurate responses. Dosu's integration with Astronomer simplifies data engineering tasks, enables regular support and maintenance, and provides a reliable foundation for building AI applications. The partnership between Dosu and Astronomer is a natural fit, as Astronomer powers Dosu's data operations, accelerating product velocity, while Dosu's new features free up the Astronomer team's time to further improve their platform, benefiting both companies.

Astronomer has significantly improved its incident management process over the past year by unifying the efforts of its Research and Development (R&D) and Customer Reliability Engineering (CRE) teams under a single framework. Initially, the lack of a cohesive process led to confusion and inefficiencies, but a new unified approach, established in February 2023, streamlined communication and collaboration during incidents. This new process includes clear incident definitions, severity levels, an Incident Manager On-Call rotation, and a focus on effective postmortems. Astronomer has also adopted the "Self Responsible Teams" philosophy, ensuring that developers support the services they build, thereby distributing on-call responsibilities more equitably and reducing the burden on infrastructure teams. A custom internal tool, Incident Buddy, was developed to automate incident management tasks, though Astronomer is considering third-party solutions for greater functionality in the future. The changes have bolstered internal trust and increased reliability for customers, with ongoing efforts to enhance automation and self-healing capabilities for further resilience.

Astronomer faced challenges in managing release notes for its Astro product due to a distributed team and a fast-paced, weekly release cycle that involved numerous changes. Initially, the process involved technical writers manually sifting through pull requests, which was time-consuming and error-prone. Attempts to streamline the process through product managers were insufficient, as they lacked insight into smaller but impactful code changes. The solution came with the introduction of Towncrier, a tool that automates changelog generation from "news fragments" added by engineers to their pull requests. By repurposing Towncrier's capabilities, Astronomer created an internal changelog that technical writers could use to draft release notes efficiently. To integrate this process smoothly into the engineering workflow, they adjusted the process to be initially optional and later mandatory, utilizing GitHub actions to extract change descriptions. This approach reduced the time needed for writing release notes from hours to minutes while fostering a stronger collaboration between engineers and technical writers and allowing for continuous iteration without burdening developers.

Integrating Apache Airflow with Azure Data Factory (ADF) offers a powerful combination for data engineering, enhancing efficiency without disrupting existing workflows. While ADF excels in creating user-friendly, low-code data jobs within the Azure ecosystem, Airflow adds orchestration capabilities, enabling complex, conditional workflows and integration with non-Azure services. This synergy allows seamless management of data pipelines across hybrid and multi-cloud environments through a single interface. In practice, Airflow can dynamically scale ADF pipelines, support data-driven scheduling, and integrate data quality checks using tools like Great Expectations, all without altering existing ADF jobs. This approach not only preserves workflow integrity but also improves data analysis and reporting, providing valuable business insights and enhancing customer satisfaction.

Astronomer has relocated its headquarters to 50 West 23rd Street in New York City to better engage with the open-source community and its growing customer base, while maintaining a presence in Cincinnati. The move provides Astronomer access to New York's vibrant tech scene and a larger talent pool, which aligns with its plans to expand its team and fill sales and engineering roles. The new office, equipped with collaborative spaces and a view of the Empire State Building, supports the company's growth and ambition to host community events, especially as Apache Airflow® downloads soar to 30 million a month. This relocation follows a year of momentum for Astronomer, marked by an expanded leadership team, an increased customer base, and several product innovations, including updates to the Astro platform and new integrations with Apache Airflow® and Microsoft Azure, enhancing data orchestration and AI capabilities. The company was also recognized as one of CRN’s Hottest Big Data Startups of 2023.

Astronomer, founded in 2018, has developed Astro, a fully managed SaaS platform that runs and manages Apache Airflow at an immense scale, processing over 100 million tasks monthly across all major public clouds. As a data orchestration platform originally built by Airbnb, Airflow's open-source nature allowed it to gain popularity, which Astro capitalizes on by offering scalable deployment solutions. Astro's architecture comprises a Control Plane and a Data Plane, with services like Stagehand and Harmony managing cluster provisioning, configuration, and maintenance. The Astro API facilitates user interactions, while the Control Plane ensures security by preventing public internet exposure. The system's observability is enhanced by Symphony and Laminar, which monitor deployment health and optimize resources through intelligent autoscaling. This inaugural blog post by Astronomer intends to kick off a series that will delve deeper into the platform's unique features and technical details.