August 2021 Summaries

Trino, previously known as Presto SQL, and Starburst are tools designed to enhance the efficiency and ease of data scientists' roles by providing faster access to data and simplifying data management. At the Datanova conference, discussions highlighted how these tools support a transition from traditional monolithic data management to a distributed Data Mesh approach, enabling seamless and direct data access. For data scientists, this means reduced cycle time in model development and the ability to ask more questions and gain deeper insights from their data. Trino's SQL-based approach allows for efficient data preprocessing, facilitating rapid exploration and analysis, which is crucial for building predictive models. The tools were initially used at Facebook for handling large data sets, particularly for marketing and financial services applications, and they help in data profiling, exploration, and reducing data preparation time through SQL-driven processes. Additionally, Trino integrates with DevOps practices without altering the basic approach, allowing operational data to be treated as source data, thereby enhancing the overall efficiency and productivity of data science tasks.

The article highlights the importance of query performance in data processing, emphasizing its impact on customer satisfaction, energy and resource efficiency, and cost-effectiveness. It discusses recent advancements in Starburst's SQL engine, particularly in the Starburst Enterprise 360-e release, which include improvements to the Parquet Reader and the introduction of enhanced dynamic filtering techniques. These enhancements have led to significant performance boosts, with Parquet reader speed increasing by up to 30% and dynamic filtering offering up to 6-fold improvements in specific queries. Additionally, the Delta Lake format has benefited from new optimizations, such as the introduction of an ANALYZE command to gather crucial statistics like the number of distinct values, which enhances query planning and execution. The article underscores the continuous efforts to further optimize performance across the entire query engine stack, aiming to reduce infrastructure costs and time-to-insight for Starburst users.

Starburst, led by Co-Founder and CEO Justin Borgman, aims to revolutionize data access and performance through a multi-act vision that progresses from Data Lakehouse to Data Mesh architectures. Initially, the company enhances Data Lake capabilities by integrating high-performance SQL engines and active data warehousing functionalities, achieving this through their collaboration with the Trino open-source project. Starburst's approach offers advanced autoscaling, fine-grained access control, and integration with major cloud providers. The second phase focuses on expanding into a Data Mesh paradigm, enabling self-service analytics with over 40 connectors to various data sources and integrations with data governance tools, while their Starburst Cached Views simplify data location abstraction. The third act introduces Starburst Stargate, facilitating cross-cloud and cross-region Data Mesh designs to optimize global data analytics and comply with data sovereignty regulations. While these stages form part of a broader five-act strategy, Starburst continues to innovate, inviting others to follow their evolving journey.

Data Mesh is proposed as a transformative approach to enterprise data management, addressing the limitations of traditional centralized data storage systems like data warehouses and data lakes, which have struggled to provide agile and trustworthy business insights. Coined by Zhamak Dehghani, Data Mesh is a decentralized, sociotechnical strategy that emphasizes domain-oriented data ownership and peer-to-peer data collaboration, allowing organizations to align technology with business domains and effectively manage analytical data. This approach aims to close the gap between analytical and operational data by dissolving fragile data pipelines and reducing the complexity of data copying. Instead of treating data as an asset, Data Mesh views it as a product, empowering generalists to develop and deliver data products without relying on centralized data engineering teams. This method promises to enhance responsiveness to change, improve data trustworthiness, and ultimately focus on user satisfaction rather than mere data volume.

Starburst aims to revolutionize the traditional enterprise data warehousing model by offering a more flexible and decentralized approach to data management. Unlike traditional systems that consolidate data into a single repository, Starburst proposes leveraging Data Lakes, which provide open data formats and separate storage from compute, to reduce costs and increase elasticity. This approach addresses the integration challenges and data silos faced by many organizations, allowing for greater data visibility and access across multiple clouds and regions while adhering to global regulatory requirements. Inspired by the Stonebraker Principle, which suggests using purpose-built databases for specific tasks, Starburst supports a diverse range of data sources, including legacy systems and new digital systems, to unlock insights and enhance data performance without vendor lock-in.

As organizations transition their analytical ecosystems to the cloud, they are increasingly adopting hybrid data storage solutions that combine traditional relational database management systems (RDBMS) with distributed data stores like S3, Azure's ADLS, and HDFS. This approach leverages the strengths of both systems, with RDBMS managing mutable data that requires frequent updates, and distributed stores handling "write-once, read-many" data such as events and IoT transactions. Modern tools and federated query engines, like Starburst's Trino, facilitate seamless data queries across these systems, offering flexibility and scalability without the data lock-in typically associated with on-premise solutions. This hybrid architecture supports a variety of emerging data patterns, including data lakes, data meshes, and multi-location data setups, enabling companies to efficiently manage and analyze vast amounts of data while mitigating downtime and quality issues during migrations.

Despite significant investments in next-generation data storage systems, traditional data warehouses and lakes have failed to deliver the agility and trustworthy insights needed for intelligent business decisions, prompting a shift towards a Data Mesh approach. This decentralized paradigm, defined by Zhamak Dehghani, focuses on managing analytical data in complex environments and is the focus of her upcoming O’Reilly book, sponsored by Starburst. The book's first chapter, "The Inflection Point," highlights the challenges organizations face, such as the need for voluminous, diverse, and up-to-date data to power analytics and machine learning, and the inefficiencies of traditional data architectures where operational and analytical data remain disjointed. Despite substantial investments, a recent survey indicates that only a fraction of firms have successfully become data-driven or are effectively competing with data and analytics, underscoring the need for new data management strategies. Looking forward, the book aims to explore innovative ways to aggregate, access, and engage with data, moving beyond the limitations of current practices.

Starburst positions itself as the analytics engine for Data Mesh architecture, offering a solution to the challenges of traditional centralized data systems like EDWs and Data Lakes. Data Mesh embraces distributed data, enabling businesses to make faster, data-driven decisions by accessing data where it resides, reducing complexity and cost by eliminating the need for data movement across domains. Starburst, built on open-source Trino, facilitates real-time access to distributed data, empowering self-service analytics across business domains without striving for a single source of truth. This approach minimizes infrastructure costs, prevents vendor lock-in, and integrates easily with existing tools, enhancing time-to-insight. Starburst's capabilities are showcased through successful implementations by companies like Zalando and Comcast, and the platform supports integration with other open technologies, reducing the total cost of ownership.

Microsoft's Azure cloud platform, particularly Azure Data Lake Storage (ADLS), has become a popular choice for companies migrating from on-premise data lakes due to its low cost and high performance. Starburst facilitates this transition by offering a high-performance, concurrent query engine that allows querying data directly from various sources like ADLS, Synapse, SQL databases, and even NoSQL systems without the need for data migration. Starburst's tools support ANSI SQL and offer features like Cached Views for enhanced performance and a Delta Lake Connector for transactional data operations, which are crucial for data compliance and updates. Additionally, the Starburst Stargate feature enables efficient query processing across data spread in different locations, including on-premises and multi-cloud environments, minimizing egress costs and improving performance. This approach helps enterprises easily manage and query their diverse data landscapes using standard SQL, making data access seamless and efficient.