Airflow in Action: From Steel City to Data City. Pittsburgh, Astro and Open Government Data

At the Airflow Summit, Alida Laney, Data Engineer for the City of Pittsburgh, presented how a small two-person team effectively utilizes Apache Airflow and Astro to modernize the city's data infrastructure and enhance operational efficiency across various departments. Faced with unique governmental challenges, including stringent vendor restrictions and the need for inter-departmental data sharing, the team transitioned from Google Cloud Composer to the fully managed Airflow service, Astro, resulting in a significant reduction in infrastructure management time. This change enabled the team to focus on building comprehensive data pipelines that support a wide range of civic needs, including open datasets used by local universities and community organizations, and tools like OneStopPGH Insights for city planning and public transparency. The automation of previously manual processes, such as cross-checking property purchase applicants against permit violations and tax records, exemplifies Astro's impact, reducing a task that once took weeks to mere minutes. This commitment to data-driven governance has earned Pittsburgh a Gold-level What Works Cities Certification, highlighting the city's ability to deliver efficient and reliable services similar to larger cities like Boston and Seattle. Looking ahead, the team plans to expand their data initiatives, including an internal data mart using Google Dataplex, while continuing to democratize data and refine government efficiency through community-driven open datasets.