August 2023 Summaries

Starburst has expanded its partnership with Google Cloud to enhance AI capabilities for users by integrating its unified data lake analytics platform with Google Cloud’s Vertex AI machine learning platform. This collaboration allows organizations to leverage AI services across multiple data sources, regardless of their location, by federating diverse datasets into machine learning models built with Vertex AI. The integration aims to accelerate development, offer flexibility, and ensure scalability for machine learning-powered analytics, with use cases spanning financial services, retail, and healthcare sectors. By using Starburst with Vertex AI, companies can optimize data exploration, model training, and deployment, thereby improving predictive capabilities and operational efficiencies.

The text provides an in-depth comparison of data warehouses, data lakes, and data lakehouses, explaining the benefits and limitations of each architecture. Data warehouses are highlighted for their structured data storage, high performance, reliability, and use of SQL, but they can be costly and require complex ETL processes. Data lakes offer a modern, versatile, and cost-effective alternative, capable of storing diverse data structures and supporting large-scale data science and machine learning applications. Data lakehouses combine the strengths of both data lakes and data warehouses, offering improved performance, reduced costs, and enhanced functionality such as ACID compliance and schema evolution. They provide a democratized access to data across organizations while maintaining the agility and low-cost benefits of cloud object storage. The text emphasizes that data lakehouses represent an evolution in data architecture, allowing businesses to reduce reliance on traditional data warehouses while optimizing data management and analytics capabilities.

Starburst Galaxy is positioned as a critical tool for enhancing security incident response programs, emphasizing the importance of rapid data analysis and collaboration to mitigate security breaches and maintain compliance with regulations. The platform offers features like query sharing via Starburst Gravity, which facilitates streamlined data investigations and collaboration among security teams during incidents, while its autoscaling capability ensures that compute resources are efficiently managed without manual intervention. Fault-tolerant execution guarantees the reliability of data analytics even under high-pressure situations, and the Data Products feature allows for preconfigured data sets to be used immediately during incidents to enhance data discoverability. Additionally, Warp Speed accelerates query processing for quick evidence collection, and audit logs provide a clear record of administrative actions to assist with regulatory compliance and insurance claims. By integrating Starburst Galaxy into incident response plans, organizations can ensure timely investigations and maintain trust with customers by minimizing downtime and adhering to compliance requirements.

Starburst has launched a new community forum designed to serve as a comprehensive knowledge repository for users of Starburst Galaxy, Starburst Enterprise, and Trino. The forum is envisioned as a hybrid between Stack Overflow and traditional web forums, providing a platform for users to ask questions, share insights, and showcase projects. It aims to foster community interaction and facilitate the exchange of best practices and advice among users of Starburst products and Trino. The forum, powered by Discourse, offers a searchable knowledge base that makes finding previously answered questions more accessible than platforms like Slack. Supported by Starburst's developer relations team, the forum includes content from a predecessor Trino forum and offers a "Showcase" category for users to highlight their innovative projects. The initiative is part of Starburst's effort to create a more communal and discoverable space for knowledge sharing and to complement existing resources like conferences and Slack channels.

Data catalogs and data products are integral components of a robust data strategy, each serving distinct but complementary roles. Data catalogs function as comprehensive inventories of an organization's data assets, enhancing data literacy by providing clarity, coherence, and secure access to data, thus aiding in metadata management, data governance, and regulatory compliance. They help users navigate vast data ecosystems by offering features like search capabilities, data lineage, and metadata management, which together foster trust and accessibility. On the other hand, data products are curated datasets designed to provide business value swiftly by being easily discoverable, understandable, and trustworthy. They are created with specific structural, process, and functional characteristics that ensure their utility and alignment with business goals, often using federated queries to minimize data movement and enhance agility. Starburst's Galaxy and Gravity platforms exemplify this synergy by combining data cataloging and product management with centralized governance, enabling efficient, secure, and value-driven data consumption. Upcoming features like data quality and lineage aim to further enhance this ecosystem by providing deeper insights into data provenance and fitness, fostering confidence and trust between data producers and consumers.

BestSecret, a European online fashion retailer, is transitioning its data architecture from Snowflake to a more open data architecture using Trino and Starburst, driven by the need to accommodate emerging projects that their existing data stack could not effectively support. The company faced challenges in managing and transforming data with their previous setup, which included dbt, Snowflake, and Airflow, leading to inefficiencies and bottlenecks in their central data engineering team. By experimenting with Trino and conducting a Proof of Concept with Starburst, BestSecret opted for an open data stack that employs the Icehouse design pattern and the Iceberg format, allowing for greater flexibility and efficiency. This new architecture eliminates the need for a monolithic data warehouse, promoting a zero ELT approach with open table formats that prevent vendor lock-in and enable custom functionalities. The shift has resulted in improved efficiencies and has introduced optionality into their data management processes, although Snowflake remains part of their ecosystem.

A GigaOm report compares the cost, time, and effort required to adopt a Snowflake cloud data warehouse versus a data lakehouse powered by Starburst, concluding that Starburst offers superior price-performance and faster time-to-insight at a lower total cost of ownership (TCO). The report highlights that while both systems achieve nearly identical cost-performance on standard BI workloads, Starburst excels with 5x better price-performance on complex and semi-structured workloads. Additionally, a Starburst lakehouse is up to 67% faster to set up than migrating data into Snowflake, reducing the overall migration effort significantly. Over a three-year period, Starburst's TCO is up to 55% less than Snowflake, owing to lower costs in infrastructure, data migration, ongoing maintenance, and opportunity costs during migration. The findings demonstrate that Starburst's modern data lake approach not only provides comparable query performance to traditional data warehouses but also offers a more flexible, cost-effective alternative that mitigates data movement and duplication challenges.

AI and data products are increasingly integrated to enhance user experiences and accelerate business solutions, driven by advancements in technology and the accessibility of tools like ChatGPT. This integration shifts the focus from data preparation to insight generation, simplifying data interaction for business users and fostering innovation. Starburst plays a key role by abstracting complex data architectures, allowing data science teams to access vast data assets for model training and enabling business teams to develop AI-driven solutions without infrastructural challenges. AI streamlines data product design and operations by automating quality checks, identifying security risks, and customizing user experiences. Organizations are encouraged to start with small teams and a limited number of data products to understand their integration into existing ecosystems before expanding, aiming for early wins to build momentum in their transformation initiatives. While AI and data products offer promising advancements, challenges in data quality, compliance, and business-focused data ontologies remain critical hurdles to address.

Organizations aiming to enhance their data-driven strategies can leverage Onehouse and Starburst to build a near real-time data lake, which combines the capabilities of a data warehouse and ingestion tool at reduced costs. The process involves integrating Onehouse's Stream Capture with Postgres, which uses technologies like Debezium, Kafka, and Apache Hudi, to efficiently ingest and manage data in a lakehouse. This data is then made accessible for analytics through Starburst by configuring an S3 catalog with an AWS Glue metastore, allowing users to perform SQL analytics. This setup, enabling faster insights and optimizing data tasks, can be implemented swiftly, allowing businesses to handle data from multiple sources and formats, thus scaling efficiently from gigabytes to petabytes.

Salesforce has adopted Trino to transform its approach to performance engineering, analytics, and anomaly detection, resulting in significant improvements in data processing speed and efficiency. Trino has become the central platform for handling large volumes of production data, allowing Salesforce teams to perform advanced queries rapidly, build near-instant reports and dashboards, and reduce analytics costs. This capability has enabled faster decision-making, proactive issue resolution, and enhanced service availability, ultimately benefiting customer experiences. Trino's integration has also allowed for extended historical data retention, improved query execution times, and the simplification of data collection processes, which were previously cumbersome with tools like Splunk. The partnership with Trino has empowered Salesforce to enhance operational efficiency, support real-time anomaly detection, and plan for future performance improvements, positioning the company to continue delivering high-quality services to its vast customer base.

Data federation is a method that allows querying and combining data from multiple, varied sources without the need to move or consolidate data into a single repository. This approach addresses the challenges of data silos and enhances business insights by connecting disparate data sources, thus providing more flexibility and reducing costs associated with data movement and storage. Starburst Galaxy, utilizing the Trino SQL query engine, exemplifies this by offering a wide range of connectors to various data sources, including cloud and on-premise systems, NoSQL stores, and relational databases. This system allows for efficient querying and integration of data across multiple platforms, improving decision-making and organizational agility. Data federation is distinguished from data virtualization by its focus on integrating data for querying, whereas virtualization includes additional services such as metadata management and data abstraction. However, implementing data federation requires addressing challenges such as maintaining data quality, optimizing query performance, and managing security protocols across varied data environments.

Starburst Galaxy's August 2023 update highlights two major product announcements: the general availability of Warp Speed and the introduction of the Galaxy Telemetry catalog. Warp Speed, an autonomous data lake acceleration layer, offers significant improvements in query speed and CPU time without additional costs, as preview customers experienced a 4X increase in speed and a 7X reduction in CPU usage. The Galaxy Telemetry catalog allows users to explore their query history and is available for private preview. The update also includes support for querying data from Apache Pinot and Apache Druid, with future support for CosmosDB, Clickhouse, and OneHouse. Additional features such as parallel mode for Snowflake catalog connections, universal search, schema discovery, catalog explorer, and column masks have been announced, enhancing the overall functionality of the Starburst Galaxy platform.