July 2021 Summaries

Data Mesh represents a modern approach to analytics at scale, addressing the limitations of traditional centralized data management systems such as the enterprise data warehouse model. Coined by Zhamak Dehghani, Data Mesh emphasizes decentralization, allowing more efficient access to distributed data and focusing on organizational, architectural, and technological improvements. It is based on four principles: domain-driven data ownership, treating data as a product, creating a self-service infrastructure platform, and federated computational governance. Unlike a single technology or solution, Data Mesh involves rethinking the human aspect of technology and adopting open data platform strategies. Starburst, leveraging its Trino-powered query engine, aligns with this approach by enabling faster access to distributed data, and offers resources to support organizations transitioning to a Data Mesh architecture.

Apache Kafka, initially developed at LinkedIn and open-sourced in 2011, serves as a scalable and fault-tolerant platform designed to optimize data streaming and support high-performance applications. It is widely used in various business contexts, such as customer 360 applications, hospitality, fraud detection, and predictive maintenance, due to its capability to manage real-time data effectively. However, integrating Kafka with query capabilities for better data understanding often requires additional tools like Solr or Elasticsearch. The addition of ksqlDB, a SQL-like streaming engine, enhanced access for users not familiar with programming languages like Java or Python. Meanwhile, Starburst Enterprise, based on Trino, offers a centralized access point for data consumers, allowing them to query Kafka topics using SQL. This is facilitated by Starburst's Kafka data connector, which simplifies the handling of Kafka's complex data structures through schema metadata and JSON functions, enabling analysts to enrich streaming data with external sources without the need for extensive data migration. By federating queries across multiple data sources, Starburst enhances the usability and accessibility of Kafka streaming data, improving the speed and efficiency of data analysis and reporting for users of all skill levels.

Data lakehouse architecture is emerging as a powerful solution that integrates the strengths of data lakes and data warehouses, offering a more efficient and flexible data management approach. Unlike traditional data warehouses, which require extensive data preparation and movement, lakehouses allow business intelligence, reporting, data science, and machine learning experts to collaborate on the same data without unnecessary data transfers. This architecture provides optionality by maintaining data in low-cost storage, supporting open data formats to avoid vendor lock-in, and enabling a data consumption layer that facilitates real-time, secure, and scalable data sharing, exemplified by the Delta Sharing protocol. The use of Delta Lake's open-source table format enhances performance by allowing time travel features, metadata exposure, and query optimizations. Companies like EMIS Health have successfully implemented data lakehouses to manage vast data volumes efficiently, particularly during the COVID-19 pandemic, demonstrating the architecture's capability to meet complex data analytics needs in real time.

Martial Coiffe and Victor Coustenoble, leading Starburst's new French subsidiary, emphasize the company's mission to simplify data access and analytics in the evolving French market. With extensive backgrounds in data management and software, both executives recognize the increasing complexity of digital transformation, driven by the proliferation of data and the need for real-time decision-making. They highlight Starburst's unique approach in offering seamless, decentralized data access through SQL and familiar analytical tools without data duplication, addressing challenges of data heterogeneity and accelerating business analytics. The French market's shift towards digitalization and cloud migration aligns with Starburst's capabilities, as companies seek agile solutions to manage their growing data needs. Both Martial and Victor stress the importance of overcoming traditional data architecture constraints, advocating for innovative concepts like Data Mesh to enhance data utilization. As Starburst continues its expansion in Europe, its flagship offerings, including the newly introduced Galaxy SaaS platform, are set to further democratize data accessibility and analysis, positioning the company as a key player in the data analytics landscape.