November 2023 Summaries

Starburst Galaxy introduces a new streaming ingest feature, currently in private preview, that allows for the continuous ingestion of data from Kafka-compliant topics into Apache Iceberg tables in near real-time while ensuring exactly-once processing guarantees. This solution simplifies the traditionally complex and costly process of building custom data ingestion systems, which often lack efficiency and scalability. Starburst's streaming ingest service offers a straightforward three-step setup, connecting Kafka sources to Iceberg tables and providing a transformation process for data mapping. This managed service is designed to handle large volumes of event data efficiently, making it a scalable and cost-effective alternative to existing solutions. Additionally, Starburst plans to enhance the service further by integrating it with materialized views systems to support the creation of declarative pipelines for data preparation.

The partnership between Starburst and Collibra aims to enhance enterprise data catalogs by integrating Starburst's metadata aggregation capabilities with Collibra's data intelligence solutions, creating a more efficient and comprehensive data marketplace. This collaboration facilitates the discovery and access of trusted data products by automating and streamlining data governance and access processes, reducing complexity, and ensuring compliance with governance policies. By combining data from multiple sources into single products and cataloging them automatically, the partnership allows faster access to high-quality data, improving data literacy and enabling strategic business insights. Plans are underway to further integrate their offerings, including advanced governance features and support for data products, to elevate the functionality and effectiveness of enterprise data catalogs.

Starburst has announced the general availability of attribute-based access controls (ABAC) in its Gravity platform, a system that allows data administrators to manage permissions based on user and data context factors, enhancing scalability and data governance. This release includes AI-powered data classification, which reduces manual tagging by suggesting relevant tags through an AI language model, thereby minimizing human error and improving data security. The ABAC framework allows tagging at various data levels, enabling the creation of policies with specified scopes, matching expressions, privileges, and optional expiration times. Additional features include real-time column masks and row filters that provide cell-level security by obfuscating or reducing data visibility based on user roles. The AI data classification currently supports around 22 PII and location-based classifiers, with plans to expand these offerings and allow for custom classifier definitions in the future.

Starburst has introduced new AI-powered features in Starburst Galaxy to enhance data lake analytics by leveraging generative AI for text-to-SQL and SQL-to-text workflows. These features aim to make data more accessible to business users by allowing them to generate SQL statements from natural language questions and providing explanations for SQL queries, thereby improving data literacy and business continuity. Although these features are still experimental and require human oversight to ensure accuracy, they represent a significant step in integrating AI into data management practices. The features are currently in public preview and can be enabled by users with the necessary account privileges, with Starburst continuing to explore ways to incorporate richer metadata for more precise results.

Starburst Galaxy introduces automated data lake optimization to enhance query performance and storage utilization in data lakes, addressing challenges posed by modern table formats like Apache Iceberg. Unlike traditional databases or data warehouses, data lakes require manual maintenance, often consuming significant time and resources from data teams. The new automated optimization in Starburst Galaxy encompasses four main operations: data compaction, profiling and statistics, vacuuming, and data retention. Data compaction consolidates smaller files for faster querying, while profiling and statistics refresh metrics for optimal query execution. Vacuuming removes orphaned files resulting from failed queries, reducing storage clutter, and the data retention feature allows users to manage snapshot storage by setting retention thresholds, addressing issues of version control and storage costs. These features aim to streamline data maintenance and are expected to be available for private preview in early December.

Starburst Galaxy, showcased at AWS re:Invent 2023, introduces new features to simplify and enhance the development of interactive data applications on data lakes. The platform aims to alleviate the complexities of data lake architecture by providing an all-in-one solution that integrates various functionalities, such as streaming ingest, automatic data classification, and secure data sharing. Streaming ingest allows real-time data import from sources like Kafka, while automatic classification uses AI to tag data, aiding in governance and security, especially for sensitive information. The platform also supports automatic data optimization and universal data sharing, which facilitate efficient data management and integration with third-party sources. Additionally, AI-powered tools like text-to-SQL processing enable business users to perform exploratory analytics independently, freeing up data teams for more complex tasks. These enhancements position Starburst Galaxy as a cost-effective alternative to traditional data warehouse models, offering scalable and efficient data lake management.

The combination of Starburst Galaxy and Ibis offers a powerful solution for building data-intensive applications by connecting cloud data sources for processing and analysis with the optimized Trino clusters of Galaxy, ultimately presenting the data to end users through Ibis's pandas-like API. This setup provides an efficient workflow for data scientists, enabling complex data manipulations and computations across various analytical backend systems. The process involves setting up a Starburst Galaxy account, connecting a data lake catalog, and using schema discovery to register and query datasets as tables, which facilitates automation of new file discovery within the data lake. The integration is demonstrated through a tutorial using NYC Taxi trip data, illustrating how to prepare, upload, and analyze datasets with this combination, leveraging Trino's fast distributed SQL query engine and Ibis's user-friendly interface.

Starburst has introduced data observability features in its Gravity platform, now in public preview, to enhance data management within its Starburst Galaxy ecosystem. These features aim to address common challenges faced by businesses, such as poor data quality and slow issue resolution, which can hinder decision-making and strategic data programs. By integrating data profiling, data quality, and data lineage capabilities, organizations can gain better visibility into their data systems, quickly identify and resolve issues, and understand the origins and downstream impacts of data problems. Data profiling generates statistics on tables, while data quality rules allow teams to set parameters for data consumption. Data lineage provides a map of data flow and transformations to help trace and rectify issues at their source. These features aim to reduce guesswork and streamline processes for resolving data quality issues, offering organizations flexibility in managing their data across various sources. Starburst plans to further enhance these capabilities in the future, promising more robust solutions to support data-driven organizations.

The artificial intelligence (AI) life cycle is a comprehensive framework comprising three main stages: data discovery, model development, and model deployment, collectively known as the "3Ds." Initially, data discovery involves identifying and obtaining relevant data, a phase that can consume up to 80% of a data scientist's time but is crucial for feature engineering. Next, the model development phase is where data scientists apply statistical and mathematical techniques to create analytical models that answer specific business questions, providing significant value. Finally, model deployment ensures that the outputs from these models are used by decision-makers, thus realizing the return on investment in AI. Tools like Starburst can streamline the data discovery phase by allowing data scientists to focus more on innovative model development, ultimately accelerating the AI project life cycle and enhancing organizational performance by enabling efficient data management across diverse sources.

Sequential and parallel processing are two distinct methods of executing software tasks, with sequential processing handling tasks one at a time and parallel processing executing multiple tasks simultaneously. While sequential processing was once the norm in general-purpose computing, parallel processing has become essential for handling large-scale data analytics due to its ability to divide tasks into smaller units and distribute them across multiple processors, significantly reducing execution time. Parallel processing is especially advantageous for big data analysis, leveraging modern multi-core processors and cloud computing to manage petabyte-scale datasets efficiently. The choice between schema-on-write and schema-on-read also affects data processing; schema-on-write requires defining data structure before processing, ensuring reliability, while schema-on-read offers flexibility by allowing data structure to be defined at query time. Starburst's open data lakehouse analytics platform exemplifies the benefits of massively parallel processing by enhancing performance and cost-efficiency for diverse data consumer needs, from business intelligence to machine learning applications.