March 2022 Summaries

Data Mesh TV discusses the shift in data access strategy from restrictive security measures to creating streamlined, democratized pathways for data consumers, emphasizing the balance between fast access and strong governance. The conversation highlights the benefits of a Data Mesh approach, which provides a single point of entry to well-defined Data Products, enabling faster and more efficient access while maintaining compliance and risk controls. By organizing data into Domains with federated governance, domain owners can autonomously manage access, reducing the complexity for consumers. Immuta plays a pivotal role in this ecosystem by offering a unified platform to manage access across diverse data sources, ensuring secure and reliable execution of access policies.

Starburst has implemented significant performance enhancements to improve query processing by focusing on increasing CPU utilization and reducing query wall time. A major change involves updating Trino’s query execution policy to default to a phased approach, which schedules only the stages of a query that can make progress, thereby reducing latency. Additional improvements include adaptively setting task concurrency to match the number of physical cores on a node and increasing the hive.split-loader-concurrency to enhance the processing of partitions and small files. Internal benchmarking showed an average 13% reduction in wall time, with improvements as high as 50% for specific queries. These enhancements are available in the upcoming LTS release and require minimal user intervention, as they leverage existing configurations to deliver faster query processing out of the box.

Starburst and Databricks have collaborated to enhance the Delta Lake connector, which has been donated to the open-source Trino project, allowing users to create and query a lake house with open-source software. Initially launched with read capabilities in April 2020, the connector has evolved to include write capabilities, data management, and performance improvements. This collaboration signifies a commitment to the open-source community, with both companies contributing to feature enhancements, performance optimizations, and issue resolutions. Starburst's VP of Product emphasizes the importance of community feedback in further improving Trino, while Databricks supports the Delta Lake open-source community by developing a standalone Delta Lake reader library to enhance performance and reliability. The partnership aims to foster a robust ecosystem of collaboration among users, customers, and contributors, with continuous integration setups and a test environment donated by Databricks to ensure software reliability.

Data Mesh and Data Fabric are emerging concepts in the data analytics industry, offering solutions to the challenges of distributed data management by shifting responsibilities from centralized teams to domain experts and utilizing automation, respectively. Data Mesh, introduced by Zhamak Dehghani, emphasizes a socio-technical approach where domain experts transform datasets into accessible products, while Data Fabric, as explained by Dr. Daniel Abadi, leverages AI and machine learning to automate data tasks, reducing dependency on central teams. This transformation requires data engineers to adopt new roles, with training and adaptation being vital for implementation. The discussion highlights the potential for these approaches to enhance data discovery and collaboration, suggesting that technology and entrepreneurial efforts will be crucial in facilitating this shift. Both concepts aim to redefine the role of data engineers, making their work more meaningful and connected to broader business outcomes, thus addressing existing bottlenecks in data management.

Data Fabric and Data Mesh represent two distinct yet complementary approaches in modern data management, each offering unique benefits to organizations seeking efficient and cost-effective data solutions. Data Fabric focuses on modernizing data integration and centralization through technology utilization and implementation design, often incorporating AI and machine learning to enhance its capabilities. In contrast, Data Mesh emphasizes a distributed architecture for building business-centric data products, allowing for decentralized governance and management of data directly within source systems. While Data Fabric demands significant technological investment, particularly in tools and platforms, Data Mesh shifts the cost focus toward services, making both approaches adaptable depending on an organization's specific needs and cloud strategies. Both paradigms have evolved from decades of data management advancements and can coexist to enhance data strategy and performance, as highlighted in Gartner's research, which explores the nuanced differences and potential synergies between them.

Adrian Estala, VP and Field Chief Data Officer at Starburst, discusses the strategic approach to initiating a Data Mesh strategy, emphasizing its business-driven nature and the potential for early success with minimal cost and effort. He highlights that starting with a clear alignment to the organization's business and data strategy is crucial, focusing on specific use cases that can yield immediate value. Estala suggests leveraging existing data governance processes and utilizing technologies like Trino to connect disparate data sources without the need for large-scale data migration, thereby allowing for decentralized, domain-oriented data management. He also underscores the importance of cultivating a digitally skilled culture that embraces self-service analytics, which can enhance agility and innovation. Governance within a Data Mesh should be a balance between risk and value, adapting to the specific needs of each domain, and a federated governance model is recommended to allow Domain Owners some autonomy. Overall, the article encourages organizations to assess their readiness for a Data Mesh and to build a roadmap that aligns with their unique digital transformation journey.

In a data-driven world where rapid access to information is crucial, maintaining a robust security posture when selecting a data access solution is essential, particularly for organizations subject to stringent data protection regulations like GDPR or HIPAA. The assessment of security elements before purchasing a solution can ensure compliance and save time. Key security considerations include verifying that the solution has undergone external audits for compliance with recognized frameworks, ensuring ongoing code scanning for vulnerabilities, and confirming regular penetration testing to address cyber risks. Additionally, integrated risk management practices, such as maintaining a risk register and having an incident response plan, are vital for continuous threat assessment. Limiting data access and sharing according to the principle of least privilege is also critical to ensuring data privacy. By thoroughly vetting providers and focusing on these security aspects, companies can reduce supply-chain risk and align with both business needs and compliance requirements.