Early Adoption of the Self-Service Data Infrastructure

Ten years ago, British Telecom (BT) implemented an early version of Data Mesh, focusing on self-service data infrastructure and data as a product, which addressed the challenges posed by big data's volume, velocity, and variety. This initiative, driven by Phillip Radley, then Chief Data Architect, involved creating a Hadoop as a Service (HaaS) platform in collaboration with Cloudera, despite limited tools, budget, and expertise. The project initially faced issues with over-governance, which inhibited user access, leading to a simplification of the governance process through the Hadoop User Group (HUG) and a streamlined four-step user onboarding process. Within four years, BT successfully managed 104 distinct data products, demonstrating scalability and resolving resource management issues. Radley shares six lessons from this experience, emphasizing the importance of a domain-agnostic team, a small launch feature set, reducing entry barriers, accessing source-aligned data products, clear terms for charging cycles, and governance auditing. Today, Radley continues to provide technical leadership in data and AI strategy at Thoughtworks, reflecting on the enduring relevance of challenges faced at BT and the growing interest and expertise in Data Mesh.