Airflow in Action: Kaiser Permanente's AI Pipelines Detecting Disease Before Clinicians Can

Kaiser Permanente's Division of Research operates complex AI and ML workloads using Apache Airflow on an entirely on-premises, Kubernetes-based infrastructure, emphasizing data privacy by avoiding cloud solutions. Lawrence Gerstley, Director of Data Science, highlights the organization’s reliance on Airflow to streamline compute resource access and facilitate extensive research workflows without requiring researchers to manage Kubernetes directly. This setup supports a wide array of AI and ML applications, including predictive models for medical conditions and text analysis for early disease detection, with Airflow orchestrating tasks across significant computational workloads. While Kaiser Permanente's approach underscores a commitment to maintaining strict control over sensitive health data, it also reflects broader trends in regulated industries prioritizing data residency and sovereignty. The organization is upgrading to Airflow 3 and exploring LLM-assisted Dag authoring to further democratize and enhance their research capabilities. Astronomer offers solutions like Remote Execution and Astro Private Cloud, catering to institutions that need to maintain data within their infrastructure while benefiting from managed orchestration services.