April 2022 Summaries

Data Fabric and Data Mesh are two prominent concepts in data management that, while often viewed as competing, can complement each other by leveraging their shared principles of eliminating data silos and democratizing access to decentralized data sources. Data Fabric utilizes artificial intelligence to integrate data sets using active metadata and machine learning, creating virtual domains for data management. In contrast, Data Mesh emphasizes decentralized management, where domain teams autonomously govern their domains and build data products closest to the business needs. Governance differs as Data Fabric employs a top-down approach with centrally managed metadata, whereas Data Mesh adopts a bottom-up approach with distributed governance aligned with the risk profile of each domain. Both approaches aim to enhance data accessibility, efficiency, and consumer adoption, and organizations are encouraged to learn from both to improve their data strategies. Modak and Starburst are two companies offering solutions and technologies to optimize data management practices, supporting the implementation of Data Fabric and Data Mesh by providing tools for automation, data integration, and efficient querying across distributed data environments.

Starburst's approach to data management centers on the concept of treating data as a product, aligning with the principles of a Data Mesh, which emphasizes decentralization and domain-oriented ownership. By introducing Data Products, Starburst provides a platform that facilitates the creation, maintenance, and consumption of data products in a self-service manner, reducing the need for complex data pipelines and centralized data teams. This model empowers subject matter experts within domains to curate and produce data products, fostering greater alignment between data engineering and software engineering. Starburst's user experience supports data producers and consumers in using SQL to interact with data products, thereby enhancing the efficiency of generating business insights and strategies. The platform enables seamless data querying across various sources, whether on-premises or in the cloud, without extensive infrastructure investment, thus democratizing access to data and enhancing the flow of information within organizations.

Metadata plays a crucial role in unlocking the potential of a Data Mesh by enabling discoverable, understandable, and trustworthy data infrastructures, yet its current use often falls short due to siloed tools and a lack of personalization for diverse data users. To address these challenges, the future of metadata should focus on creating intelligent, federated data management systems that integrate seamlessly with daily workflows and provide personalized experiences. This shift involves moving from top-down governance to a democratized model where metadata facilitates automated and orchestrated processes, enhancing data product health and discoverability. Implementing active metadata practices in a Data Mesh requires establishing data product shipping standards, automating processes for self-service infrastructure, and fostering a cultural change through human-driven standards and rituals. By doing so, organizations can create a dynamic action layer that powers fundamental concepts of the Data Mesh, ultimately leading to improved data accessibility and trust.

Starburst has made it easier to deploy its Enterprise analytics engine on Amazon Web Services (AWS) with the introduction of Starburst Enterprise for Amazon Elastic Kubernetes Service (EKS) in the AWS Marketplace. This deployment option streamlines the installation and configuration process by allowing users to deploy a fully optimized version of Starburst on a Kubernetes cluster using EKS's native features like autoscaling and Spot instance integration. The deployment is self-service and offered as a pay-as-you-go model, with billing managed through the user's existing AWS account. The blog post outlines a detailed setup process, including prerequisites like an AWS account and specific tools such as Helm and eksctl, and provides instructions for setting up an EKS cluster, configuring the environment, and installing Starburst. With this new offering, users can leverage the full capabilities of the Starburst platform, integrate it with AWS data sources like AWS Glue, and manage it efficiently within their cloud infrastructure.

Starburst has partnered with Google Cloud to enhance Google BigQuery's customer experience by enabling hybrid and cross-cloud data federation, allowing organizations to perform analytics across various data centers without moving the data. This collaboration offers access to data from multiple sources like Oracle, SAP, and Teradata, with over 45 enterprise connectors, all while maintaining cost-effectiveness and security. Starburst is available on Google Cloud's Marketplace with products like Starburst Enterprise and Starburst Galaxy, providing an intuitive web interface for querying data. A new offering includes a white-glove setup and a two-week trial period for organizations to set up data federation between BigQuery and other data sources, enabling them to deploy federated analytics quickly with no upfront costs and transition to a pay-as-you-go model. The initiative aims to simplify data access and discoverability, eliminate data silos, and support digital transformation by enabling advanced analytics through seamless data integration.