May 2022 Summaries

Ten years ago, British Telecom (BT) implemented an early version of Data Mesh, focusing on self-service data infrastructure and data as a product, which addressed the challenges posed by big data's volume, velocity, and variety. This initiative, driven by Phillip Radley, then Chief Data Architect, involved creating a Hadoop as a Service (HaaS) platform in collaboration with Cloudera, despite limited tools, budget, and expertise. The project initially faced issues with over-governance, which inhibited user access, leading to a simplification of the governance process through the Hadoop User Group (HUG) and a streamlined four-step user onboarding process. Within four years, BT successfully managed 104 distinct data products, demonstrating scalability and resolving resource management issues. Radley shares six lessons from this experience, emphasizing the importance of a domain-agnostic team, a small launch feature set, reducing entry barriers, accessing source-aligned data products, clear terms for charging cycles, and governance auditing. Today, Radley continues to provide technical leadership in data and AI strategy at Thoughtworks, reflecting on the enduring relevance of challenges faced at BT and the growing interest and expertise in Data Mesh.

Data-driven organizations are embracing digital transformation to harness new and existing digital capabilities, with an emphasis on integrating domain-driven architectures, multi-vendor sourcing, cloud services, product-centric teams, DevOps delivery, continuous release models, and self-service support. This evolution aims to merge these components into a cohesive digital operating model, focusing on value delivery by abstracting architectural complexity. Data Mesh is highlighted as an effective strategy that aligns well with digital transformation goals, offering benefits such as product-centric team empowerment, enhanced agility without centralizing data, and self-service models enabling autonomy within domains. The discussion underscores the importance of frameworks like IT4IT, NIST, TOGAF, and methodologies such as DevOps and Scrum to construct a resilient digital operation model that supports business needs and accelerates digital strategies.

The blog post, written by Monica Miller, discusses the evolving landscape of data processing methods, particularly the roles of ETL (Extract, Transform, Load) pipelines and interactive query engines like Trino in contemporary data management. It highlights that while ETL is traditionally fundamental for data transformation within data warehouses, it is increasingly seen as a cumbersome approach for addressing business queries that can be swiftly answered by interactive query engines. These engines offer rapid, self-service insights by allowing direct data querying across multiple sources, contrasting with the potentially slow ETL process. The post argues for the complementary use of both methods on a single platform, given that each has its unique advantages—ETL for reliable, automated data integration, and interactive engines for quick, exploratory analytics. It also introduces Project Tardigrade within Trino, aimed at enhancing query failure recovery, thereby offering a more integrated approach that combines the strengths of both ETL and interactive querying to improve efficiency, reduce costs, and increase the reliability of data pipelines.

Starburst Galaxy's new Great Lakes connectivity feature provides seamless integration with various data lakehouse file and table formats such as Hive, Delta Lake, and Apache Iceberg, enhancing data accessibility and performance without requiring users to manage underlying complexities. This functionality allows users to configure object storage catalogs like Amazon S3, Azure Data Lake Storage, or Google Cloud Storage, while Starburst Galaxy manages the rest, enabling the use of open table formats to improve analytics and reduce data movement costs. With the ability to interact with data lakes using SQL, the platform offers significant performance improvements over traditional formats, utilizing advanced features like data skipping and improved partition handling. Starburst Galaxy, a fully managed platform available on major cloud providers, ensures that users can effortlessly leverage these capabilities while being adaptable to new file types and formats in the future.