Company
Date Published
Author
Matthew Keep
Word count
719
Language
English
Hacker News points
None

Summary

Rakuten Kobo, a Canadian company acquired by Rakuten in 2012, has effectively leveraged Apache Airflow to support its diverse teams, including data science, finance, marketing, and customer support, by maintaining a single shared environment. At the 2024 Airflow Summit, Spencer Tollefson, Team Lead Data Engineering at Rakuten Kobo, detailed their strategy for enabling team self-service, which includes delineating responsibilities, establishing guardrails for new developers, and implementing scalable monitoring and access control systems. The company's approach has minimized bottlenecks and improved collaboration by clearly defining roles, such as data engineers maintaining the environment and business developers focusing on DAG creation with support. Kobo's system includes comprehensive documentation, local development environments, and staging for realistic testing, while their alert system ensures rapid response to failures. Looking forward, Kobo plans to explore YAML-based DAG authoring and multitenancy features in Airflow to further enhance scalability and separation between teams, aligning with upcoming advancements in Airflow 3.0 for improved security and task isolation.