Airflow in Action: Powering Investigative Journalism at the FT with Orchestration and AI

Apache Airflow plays a crucial role in the Financial Times' innovative approach to investigative journalism by enabling the Storyfinding team to transform vast amounts of unstructured public data into actionable insights. This is achieved through the orchestration of AI-assisted data pipelines that integrate traditional data processing with machine learning and Retrieval Augmented Generation (RAG) techniques. The team demonstrated three use cases: uncovering UK politicians' financial interests, analyzing US SEC filings for trend discovery, and monitoring US government spending, each showcasing Airflow's ability to streamline complex data workflows and enhance discoverability. By automating the structuring and linking of datasets, Airflow empowers journalists to quickly access and analyze information, leading to impactful stories that may otherwise remain hidden. The session at the Airflow Summit highlighted the importance of Airflow in maintaining repeatability, resilience, and collaboration between engineers and journalists, while also promoting rapid experimentation and adaptability as data contexts evolve.