January 2018 Summaries

Astronomer, a data engineering company, navigated frequent changes in its product direction and market focus, which resulted in a chaotic organizational structure within its GitHub repositories. To address this, the company leveraged Apache Airflow, a data workflow management system that allows developers to schedule, deploy, and monitor data pipelines as directed acyclic graphs (DAGs). By using Airflow, Astronomer could efficiently clean and organize its GitHub data, transferring it to Amazon S3 and then to Redshift for further analysis. This process involved custom Airflow hooks and operators to interact with external systems like the GitHub API, making data management more streamlined. The company also utilized tools like Prometheus and Grafana for monitoring and visualization. Through this setup, Astronomer improved its internal reporting and created dashboards for visualizing data, ultimately showcasing the effectiveness of Airflow in resolving complex data management challenges.

In a podcast interview with Maksim Pecherskiy, San Diego's Chief Data Officer, the innovative application of Apache Airflow in the public sector is explored, highlighting how it's revolutionizing data management and operational efficiency within the city's departments. Maksim's journey into government technology was influenced by his Code for America fellowship, which demonstrated the impactful use of technology in public administration. Tasked with managing data across 36 departments, he transitioned from using FME to Apache Airflow to automate data processes, citing its advantages in version control, dependency management, and scalability. His team has leveraged Airflow to enhance data transparency, automate reporting, and perform geospatial data enrichment, thereby saving time and resources while improving decision-making processes. Maksim emphasizes the importance of a culture that fosters continuous learning and adaptation to new technologies, which is supported by San Diego's forward-thinking IT infrastructure.