July 2023 Summaries

Data lake analytics are increasingly popular among data-driven companies, but managing the vast volume of data to optimize performance and cost-effectiveness remains a challenge. While partitioning strategies, such as z-ordering and clustering, can reduce data scanning, they often fall short due to the dynamic nature of query patterns and the need to filter across multiple columns. On average, 80% of compute resources are spent on ScanFilter operations, indicating that current partitioning methods are inadequate. Additionally, excessive partitioning can lead to data skew, long query response times, and degraded performance. To address these issues, Starburst offers a smart indexing solution called Warp Speed, which uses nanoblock indexing to dynamically create efficient, multi-dimensional indices without altering existing data layouts. This approach allows companies to maintain their current partitioning strategies while significantly improving query performance across diverse workloads. Warp Speed can be easily implemented through the Starburst Galaxy platform, providing an accessible way for organizations to enhance their data lake analytics.

Isaac Obezo, a staff data engineer at Starburst, discusses the implementation of Starburst Galaxy and dbt in building resilient telemetry data pipelines. He contrasts this modern approach with traditional data warehousing, highlighting challenges such as pipeline fragility and maintenance complexity. Starburst Galaxy, a fully managed data lake analytics platform, serves as a non-destructive, abstraction layer across various data sources, facilitating a more flexible and efficient data organization process. By using dbt alongside Galaxy, Starburst's team can build, test, and document models more effectively, enhancing data quality and enabling a domain-oriented workflow. This combination allows for increased scalability, agility, and autonomy, as it reduces the dependency on multiple systems and focuses on a single development lifecycle, making it easier to iterate and improve analytical models. Through this integration, the team shifts from maintaining systems to building new projects, fostering a data-driven culture that can adapt to change without significant disruption.

The text discusses the integration of Starburst Galaxy with Apache Airflow to create resilient and efficient data pipelines, leveraging fault-tolerant execution capabilities to enhance reliability and speed. By executing a Directed Acyclic Graph (DAG) with tasks that include querying data using the Starburst Galaxy engine, printing the number of records via Python, and performing a data quality check with the SQLColumnCheckOperator, the process demonstrates the seamless management and monitoring of data workflows. The blog outlines the practical steps for setting up and running this environment, emphasizing the improvements in handling ETL workloads through advanced resource-aware scheduling and granular retries. It encourages users to explore the powerful combination of Starburst Galaxy and Airflow for scalable and reliable data solutions.

Data lakehouses have evolved from traditional data lakes, offering enhanced capabilities by integrating modern table formats like Apache Iceberg, Delta Lake, and Hudi, which enable advanced querying and data processing. The architecture typically involves a three-part data pipeline comprising the Land, Structure, and Consume layers. Data begins in its raw state in the Land layer, is transformed and validated in the Structure layer, and finally becomes consumable in the Consume layer for use in business intelligence tools and data products. This process can utilize batch or streaming ingestion methods, with technologies like Kafka, Flink, and Apache Spark facilitating these operations. The use of SQL for data normalization, validation, enrichment, and technical transformation is crucial in constructing these layers, with tools such as Starburst Galaxy and Starburst Enterprise providing integration and support throughout the pipeline. Data consumption is primarily through queries, business intelligence tools, and curated data products, allowing for enhanced data visibility, discovery, and governance.

Starburst Enterprise Platform (SEP) 422-e STS is the latest release following the 420-e STS version, built on Trino 422. This update introduces new enterprise features and enhancements, incorporating developments from the recent Trino 421 and Trino 422 releases.

In the healthcare and life sciences sectors, there is a growing emphasis on utilizing real-world evidence (RWE) to enhance clinical decision-making and patient care, addressing the limitations of traditional randomized control trials (RCTs) due to their high cost and time demands. The digitization of healthcare data, driven by industry consolidation and regulatory mandates, has led to an expansive pool of real-world clinical data, which can be leveraged to improve patient outcomes and accelerate drug development. Organizations like the Mayo Clinic and EMIS Health are using extensive patient records to provide more personalized care, while companies like GSK and 23andMe are integrating RWE into clinical research to optimize trial protocols. Despite the promise of RWE, challenges such as data privacy, security, and the need for a single access point to disparate data sources remain. Solutions like data federation and data products are proposed to streamline access and use of RWE, while secure access control and data harmonization standards aim to address compliance and interoperability issues. Empowering clinicians and researchers to utilize and contribute to comprehensive data repositories is seen as crucial for closing the evidence gap in healthcare, ultimately driving more informed, real-time, data-driven decisions without necessitating new technical skills.

The article explores how to utilize dbt and Trino with Iceberg for implementing change data capture (CDC) on a data lake, specifically using Amazon DMS data stored in CSV format on S3. The process involves creating an external table to read the data, followed by developing a model named stg_dms__products that employs dbt's incremental materialization to process only new CDC records. The article outlines the use of common table expressions (CTEs) for handling insert, update, and delete operations, and discusses strategies for implementing soft deletes and hard deletes. Key techniques include generating a MERGE statement for efficient data updates and applying incremental strategies to enhance performance. Additionally, it advises on configuring Iceberg table properties and using post_hooks for operations like expiring snapshots. The article provides practical examples and configurations for these processes, and the complete dbt project is available on a GitHub repository.

Starburst is exploring the integration of large language models (LLMs) into its data exploration tools, aiming to develop a natural language interface that allows users to query organizational data without needing extensive SQL knowledge. Despite the transformative potential of LLMs, exemplified by their ability to perform tasks like converting queries or identifying tables with personal data, there are concerns about accuracy and reliance on flawed outputs, which have led Starburst to delay releasing a customer-facing product. The company has been experimenting with various projects, such as a prototype that translates natural language queries into SQL and provides an audit trail for transparency. While this technology could broaden Starburst’s user base by simplifying data access, current limitations and the need for precision mean SQL will likely remain the preferred tool for complex queries among experienced users. The ongoing research and development, fueled by successful internal hackathons and customer deployments, suggest exciting future announcements in this space.

In a blog post by Russell Christopher, the integration of Starburst, Iceberg, and Tableau is explored as a means to resolve data anomalies and enhance data analysis through a concept referred to as "time travel." The post details an instance where unexpected changes in data on a familiar dashboard prompted a deeper investigation using these tools. By leveraging Iceberg's capability to view historical data states and utilizing custom SQL in Tableau, the author was able to identify and correct the anomaly, tracing it back to a specific change made in the data. The process involved using Iceberg's time travel feature to navigate through historical snapshots of the data and the rollback procedure to restore the data to its original state. This approach serves as a troubleshooting method rather than a primary querying mechanism due to potential performance trade-offs. The article also introduces the public preview of Warp Speed in Starburst Galaxy and encourages readers to experiment with creating their own Iceberg data lakes using Starburst's platforms.