How Flatiron Health Orchestrates Cancer Research Data Pipelines

Flatiron Health, a company focused on transforming cancer care infrastructure, faced challenges with managing multiple custom data pipelines required for research projects, which strained their engineering resources and limited scalability. To address this, they adopted Prefect, a workflow orchestration tool, to streamline their data generation platform, allowing non-technical team members to manage pipelines independently and significantly reducing incident resolution time. Prefect enabled Flatiron to automate data processing tasks, improve visibility for non-technical staff, and enhance operational efficiency, resulting in substantial time and cost savings. The tool's Python-native approach simplified development and maintenance, while its architecture ensured data security, crucial for handling sensitive patient information. This transformation has inspired other teams within the organization to consider adopting Prefect for their workflow needs, demonstrating its effectiveness in orchestrating complex data operations.