Airflow in Action: Insights From Scaling Multiple Workloads Across A Shared Environment at Rakuten Kobo

Rakuten Kobo, a Canadian company acquired by Rakuten in 2012, has effectively leveraged Apache Airflow to support its diverse teams, including data science, finance, marketing, and customer support, by maintaining a single shared environment. At the 2024 Airflow Summit, Spencer Tollefson, Team Lead Data Engineering at Rakuten Kobo, detailed their strategy for enabling team self-service, which includes delineating responsibilities, establishing guardrails for new developers, and implementing scalable monitoring and access control systems. The company's approach has minimized bottlenecks and improved collaboration by clearly defining roles, such as data engineers maintaining the environment and business developers focusing on DAG creation with support. Kobo's system includes comprehensive documentation, local development environments, and staging for realistic testing, while their alert system ensures rapid response to failures. Looking forward, Kobo plans to explore YAML-based DAG authoring and multitenancy features in Airflow to further enhance scalability and separation between teams, aligning with upcoming advancements in Airflow 3.0 for improved security and task isolation.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.