Airflow in Action: Data Engineering Insights from Circle, Managing $10s of Billions Across Multiple Blockchains

Apache Airflow was successfully used by Circle, a digital financial technology company, to manage their blockchain data orchestration and scale their data platform. Circle initially implemented CI/CD practices, automated testing frameworks, and enhanced data quality checks to overcome challenges in their managed Airflow environment. Despite these improvements, they realized the limits of performance with Managed Workflows for Apache Airflow (MWAA) and began exploring alternative managed Airflow solutions like Astro from Astronomer.io. Circle is now evaluating Astro due to its support for the Kubernetes Executor, which offers flexibility in using containerized Airflow environments, as well as features such as seamless local development, streamlined environments, and data observability with Astro Observe.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.