January 2023 Summaries

Datanova 2023, a virtual and global event held on February 8th and 9th, invites data professionals who embody the spirit of rebellion, drawing inspiration from characters like Rey and Princess Leia from Star Wars. The event aims to celebrate and encourage data rebels who challenge conventional norms to create simple, efficient, and effective data platforms, promoting data freedom and democratization within organizations. Notable speakers include Zhamak Dehghani, Benn Stancil, Martin Casado, Joe Reis, Chad Sanderson, Teresa Tung, and Martin Traverso, among others, who will share their insights and experiences. The event is a rallying point for those passionate about the transformative power of data, offering opportunities to learn, connect, and be inspired in their quest for innovation and progress in the data landscape.

In a discussion on data mesh transformations, experts Adrian Estala and Maria José Lopez emphasize the importance of focusing on user experience, abstracting technology complexity, and maintaining effective communication to ensure successful implementation. They highlight that the primary goal of a data mesh is not merely to construct it but to enable businesses to efficiently leverage data assets for tangible business outcomes. Key strategies include creating data products as a starting point, which facilitates ease of access and insight without centralizing data, and implementing a federated governance model that balances accessibility with appropriate controls. Effective internal communication plays a crucial role in aligning stakeholders and fostering trust, while linking data value to business results can drive executive support. The conversation underscores the need for a flexible approach to governance and the importance of adapting to the evolving needs of data consumers and producers to sustain momentum and innovation in data mesh initiatives.

Lyft has successfully integrated Trino into its data platform for large-scale ETL processes, transitioning from Hive to achieve significant improvements in performance, cost-efficiency, and reliability. Trino, initially used for smaller ad hoc queries, now supports Lyft's extensive data operations, handling 2.5PB of daily read data and 60TB of write data across 60,000 daily queries. The transition involved challenges such as the noisy neighbor problem and frequent releases, but resulted in notable reductions in ETL runtimes, with some jobs dropping from five hours to just 20 minutes. By migrating to AWS Graviton Instances, Lyft achieved a 10% reduction in hosting costs, while autoscaling enhanced ETL efficiency by dynamically adjusting cluster capacity based on demand. Future plans include improving reliability, exploring sharding at the orchestration layer, enabling fault-tolerant execution, and potentially adopting Tardigrade, reflecting the team's satisfaction and ongoing commitment to optimizing their data infrastructure with Trino.

Datanova 2023 is set to feature a highly anticipated debate on the relevance and future of Data Mesh in the field of data engineering, with key figures like Joe Reis and Andy Mott engaging in a spirited discussion moderated by Colleen Tartow. The event, scheduled for February 8th and 9th, will explore differing perspectives on whether Data Mesh is merely a passing trend or a vital solution for modern data challenges. Hosted by Starburst, the virtual and free event promises to delve into the philosophical and practical aspects of data architecture, inviting global participants to join and potentially reshape their views on data management practices.

Datanova 2023 is an event that explores the future of data and analytics, addressing the cultural, organizational, and process challenges of becoming data-driven organizations rather than just technological ones. Key sessions include a panel discussion on the evolution of data mesh over the next decade, featuring experts from Comcast, NatWest Group, and Priceline, alongside a Boston Consulting Group study on maximizing data value, presented by industry leaders from BCG, Red Hat, and Starburst. Additionally, a fireside chat will feature influential figures from Andreessen Horowitz, Thoughtworks, and Accenture discussing modern data strategies, team evolution, and the integration of human and technological elements in data management. The event invites global participation and registration is free, offering data professionals insights into the anticipated dynamics and hurdles in the data landscape.

The text details a tutorial on building a federated data lakehouse using Starburst Galaxy, emphasizing the importance of efficiency and optimization in both personal resolutions and professional data management in 2023. The author, Monica Miller, shares her personal experience with New Year's resolutions, relating it to the current economic climate where organizations strive for optimized processes. The tutorial focuses on integrating Pokémon Go data stored in S3 with additional information from a MongoDB database, nicknamed PokéMongoDB, to create meaningful insights. The process involves setting up schemas, tables, and a reporting structure in Starburst Galaxy, enhancing data architecture through features like role-based access control and autoscaling. The tutorial serves as a practical guide supported by a GitHub repository, and the author invites readers to engage with the content and attend the Datanova conference for further learning opportunities.

The article emphasizes the growing importance of Data Products in organizations aiming to optimize their digital investments, as highlighted by a Gartner survey of over 2,000 CIOs. It discusses how Data Products, as part of a broader Data Mesh strategy, can provide competitive advantages by ensuring that the right data is delivered in the right form to the right people. The article identifies four key levers for optimizing Data Products: cost, performance, risk, and user experience, which must be balanced according to strategic priorities and trade-offs. Effective execution across the lifecycle of Data Products, consumer adoption through simplicity, commercial value generation by optimizing differentiation, and ongoing agility in optimization are highlighted as crucial factors for success. The article also underscores the significance of aligning technology, processes, and people within a comprehensive strategic plan and suggests that organizations at the initial stages of Data Mesh implementation should focus on setting optimization priorities and understanding optimization capabilities before embarking on large-scale optimization efforts.

Starburst and Immuta have collaborated to simplify cloud storage governance, addressing challenges faced by analysts and data engineers in accessing and securing data across various cloud environments. By integrating Starburst's high-performance MPP query engine with Immuta's Data Security platform, organizations can efficiently access, query, and secure data without being locked into specific vendor paradigms. This partnership allows users to access data stored across different platforms, such as AWS S3, ADLS, and Google Cloud Storage, using familiar ANSI SQL standards, thereby reducing costs and complexity. Immuta facilitates the management of access and security policies, minimizing the maintenance burden and security risks, while Starburst functions as an abstraction layer that eliminates the need for multiple dedicated connections to data sources. This integration streamlines data access, ensuring that only authorized users can retrieve sensitive information and apply necessary restrictions like data masking, ultimately reducing overhead and enhancing operational efficiency.

The concept of Data Mesh, introduced by Zhamak Dehghani in 2019, involves decentralizing data management through domain-driven ownership and federated governance, which could significantly alter the role of the Chief Data Officer (CDO). Traditionally, the CDO is responsible for data governance, data quality, and leveraging data for business value, but Data Mesh proposes distributing these responsibilities across individual domains within an organization, potentially diminishing the traditional central role of the CDO. Although some argue that the decentralization could lead to gaps in policy enforcement, others suggest that a senior leadership role is still necessary to maintain enterprise-wide data governance and drive cross-domain efficiencies, possibly evolving the CDO role into a Chief Growth Officer (CGO) focused on strategic innovation and business growth. Despite the transformative potential of Data Mesh, it is suggested that organizations will still require a senior data leader to oversee data governance and strategic data utilization.