June 2023 Summaries

Starburst Enterprise platform (SEP) 420-e STS, a successor to the 413-e LTS release, introduces new enterprise features and enhancements based on Trino 420, alongside recent Trino updates. Noteworthy features in this release include expanded managed statistics support for various connectors such as Greenplum, IBM Db2, Netezza, Redshift, Synapse, and the Starburst Stargate connector, which improves query optimization by collecting data source statistics. Additionally, SEP 420-e introduces SCIM user and group synchronization as a public preview, integrating seamlessly with existing identity infrastructures and supporting synchronization with Microsoft Azure Active Directory, thereby streamlining user and group management across access control systems.

Starburst Galaxy is a distributed SQL query engine that allows organizations to perform efficient data analysis across various sources, including data lakes, warehouses, and databases, through a unified interface. The platform supports numerous data connectors and federated queries, enabling users to perform ad-hoc SQL queries and gain insights without complex data integration or preparation. The guide provides detailed instructions on connecting clients like DBeaver, PowerBI, and Tableau to Starburst Galaxy, emphasizing the ease of accessing and exploring large data sets. It includes steps for locating connection information within the Starburst Galaxy cluster and configuring connections in each client, highlighting the platform's capacity for seamless integration and data exploration.

Starburst Galaxy's inaugural Launch Week introduced a series of new features and enhancements aimed at improving data performance and accessibility across various platforms. Central to the announcements were the public previews of Gravity, a universal data discovery, governance, and sharing layer, and Partner Connect, a portal for integrating key partner connections like AWS QuickSight, dbt Cloud, and Tableau Cloud. The introduction of cross-cloud analytics promises to simplify data analysis across clouds, while Great Lakes connectivity offers seamless interactions with modern table formats in data lakes. The Warp Speed feature, in public preview, enhances interactive workloads through smart indexing and caching, providing a 40% performance boost. Additionally, the platform now supports Graviton3 instances in four AWS regions, delivering a 30% improvement in workload performance. Starburst Galaxy positions itself as a comprehensive data lake analytics platform with these innovations, offering users fine-grained governance, enhanced connectivity, and optimized performance.

Starburst Galaxy, a fully-managed data lake analytics platform, has integrated AWS Graviton3 to enhance performance, offering up to a 30% boost in workload efficiency for its Standard cluster. This update allows analytics teams to focus on deriving insights and fostering collaboration without the burden of managing infrastructure. By promoting open data architectures and avoiding vendor lock-in, Starburst Galaxy supports innovative data workloads while reducing costs. The platform is now available in several AWS regions, including US-East (Ohio and N. Virginia) and US-West (Oregon and N. California), with plans for further expansion as Graviton3 becomes globally accessible. Users are encouraged to explore the platform with a $500 free trial to improve their data lake initiatives.

Starburst Galaxy is a SaaS analytics platform designed to augment and optimize cloud data warehouses (CDW) by providing flexibility and control over data management and costs. Unlike traditional CDWs like Snowflake, which lock users into specific technologies and can incur high costs, Starburst Galaxy offers an open data lake architecture allowing users to freely choose processing engines while federating data in real-time from various sources using numerous connectors, including those for relational and non-relational databases. This capability reduces the need to copy data into data warehouses, enhancing agility and lowering development costs. Starburst Galaxy supports data tiering to manage costs and service level agreements effectively and enables users to perform standard database operations, such as joins and updates, on data lake tables. Built on the open-source Trino project, it is capable of handling large-scale concurrent users and is used by globally recognized companies. The platform also offers Starburst Academy, which provides free online courses to help users deepen their understanding of Starburst and its ecosystem.

Warp Speed, now available in public preview in Starburst Galaxy, is an autonomous caching and indexing solution designed to enhance data teams' performance by accelerating interactive workloads up to 40%. It uses patented technology to automatically identify and cache frequently accessed data, optimizing data lake performance without requiring manual data partitioning strategies. This feature simplifies data operations for teams lacking time or expertise in query optimization, abstracting complex details from end users. Warp Speed significantly improves query execution times by minimizing full table scans and offloading resource-intensive operations, demonstrated by a 5x performance boost and a 10x reduction in CPU time in benchmark tests. Ideal for multi-dimensional data scenarios such as fraud detection and geospatial analytics, Warp Speed allows teams to either speed up workloads or expand data querying capabilities without increasing latency. Accessible through the Galaxy platform, it currently supports AWS S3 and Tabular catalogs and offers insights into index and cache usage for users to evaluate its impact.

Starburst Galaxy has announced the public preview of data products, allowing users to create and manage single- and multi-source data products with minimal data movement, enhancing dataset discoverability and self-service analytics capabilities. A data product in Starburst Galaxy acts as a container comprising business and technical metadata pointing to underlying datasets, enabling data teams to streamline analytics processes and reduce overhead. Users can either promote existing schemas to data products or create new data products directly from queries using the Trino query engine, which supports federating data across multiple sources. The platform's built-in access controls ensure fine-grained security, allowing organizations to dictate who can access specific data products, facilitating a secure and efficient distribution of high-quality datasets. This initiative aims to improve the speed of business operations by fostering a self-service data exchange between producers and consumers, with ongoing enhancements anticipated in the future.

Starburst Galaxy introduces first-class support for open table formats like Apache Iceberg, Delta Lake, and Apache Hudi, allowing users to select the most suitable format for specific workloads rather than being confined to an organization-wide decision. This approach emphasizes flexibility and adaptability, acknowledging the complexity of data environments that involve streaming pipelines, batch jobs, and ad hoc queries simultaneously. Starburst Galaxy's Great Lakes connectivity simplifies interactions with various table formats and file types, enabling seamless data operations across platforms such as Amazon S3, Azure Data Lake Storage, and Google Cloud Storage. This feature is designed to enhance user experience by abstracting technical details, thus allowing users to focus on querying their data efficiently.

Starburst recently held its annual hackathon, Hack-a-Trino, where the team explored the integration of generative AI into data engineering processes, aligning with the current excitement around OpenAI. Three notable projects emerged: Automatic Data Classification, which leverages OpenAI’s LLM APIs to predict and tag data content for improved data management; Trino AI Functions, which uses ChatGPT and Hugging Face to translate natural language into complex SQL aggregations for advanced data analysis, including tasks like sentiment analysis and fraud detection; and No Code Querying with ChatGPT, designed to simplify data querying for business analysts by allowing natural language inputs, thus advancing Starburst's goal of data democratization. Additionally, a partner project with Aderas explored using Starburst and ChatGPT for cybersecurity analytics. Through these initiatives, Starburst gained insights into the rapid prototyping capabilities of AI models, recognizing their potential and limitations in generating accurate SQL queries, while also emphasizing the importance of data governance and security in cloud environments.

Starburst Galaxy introduces cross-cloud analytics, enabling businesses to discover, access, and manage diverse data sources across different cloud platforms, addressing challenges posed by multi-cloud environments like data fragmentation and complex integration post-mergers. With the ability to enhance internal data with third-party sources, the platform simplifies data consolidation and provides a unified view without the need for time-consuming data centralization. Utilizing the Trino query engine, Starburst Galaxy supports efficient large-scale data processing, offering scalability and flexibility essential for cloud migrations. This solution not only enhances data engineering productivity but also streamlines cloud migrations by ensuring data integrity and security, eliminating the need for complex data transfer methods.

Tableau and Starburst have collaborated to simplify business intelligence on modern data lakes by introducing a seamless integration between Starburst Galaxy, a data lake analytics platform built on Trino, and Tableau Cloud, a cloud-based analytics platform. This integration allows for a one-click connector, eliminating manual connectivity tasks and enhancing productivity for data engineers. With Starburst Galaxy, users can discover, govern, and analyze data from multiple sources within a fully-managed platform, while Tableau Cloud provides an intuitive interface for data visualization and analysis. The collaboration aims to streamline workflows, improve performance and scalability, and ensure robust data governance and security, enabling users to efficiently create interactive dashboards and derive actionable insights from their data.

Google Looker and Starburst Galaxy have partnered to enhance the capabilities of business intelligence by allowing teams to extend Looker's analytics beyond Google Cloud services like BigQuery to other cloud data sources, including AWS and Azure. This integration empowers businesses with multi-cloud environments to perform cross-cloud analytics seamlessly, reducing egress costs and optimizing infrastructure investments. By leveraging Starburst Galaxy as the compute layer, organizations can now explore data at its source before deciding what to migrate into central data stores, thus improving data performance and return on investment. The integration facilitates a modern data lake approach, enabling efficient data analysis across different cloud platforms without significant expenses, while also simplifying the process of getting started with connection steps in Looker's UI.

Starburst Galaxy and Zing Data offer a modern data experience by enabling users to query and visualize data across various platforms, including mobile devices, using advanced tools like OpenAI's language models. Starburst Galaxy serves as a data lake analytics platform that centralizes disparate data sources, allowing data teams to access and govern data seamlessly across cloud databases, data lakes, and legacy warehouses. Zing Data enhances this functionality with its intuitive mobile app, empowering users to ask questions in natural language, create visualizations, and collaborate in real-time. This integration supports various querying methods, such as drag-and-drop, SQL, and natural language queries, making data accessible and actionable for everyone in an organization, from data engineers to business users. The platform's use cases include enabling convenience store chains to optimize product mixes, SaaS companies to centralize analytics, and real estate firms to monitor transactions on the go, showcasing its versatility and real-time decision-making capabilities. Both platforms are cloud-native and offer free trials, providing an easy setup for users to start leveraging their data more effectively.

Starburst Galaxy is introduced as a data lake analytics platform designed to efficiently analyze large and complex datasets within cloud data lakes. Launched with the aim of simplifying the use of Trino without requiring teams to specialize in it, Starburst Galaxy addresses the common issue of data silos that emerge despite efforts to centralize data in lakes or warehouses. The platform provides a single point of access to cloud data sources, an analytics engine for executing workloads, and a component named Gravity for data discovery, governance, and sharing, all while allowing users to maintain control over their data storage. Users have reported significant improvements in query execution times and operational scalability, with the platform's flexibility being highlighted as a key advantage over other vendors. The company offers a free trial and invites participation in an event to explore new features and tutorials.

Gravity, a new feature in Starburst Galaxy, offers a unified solution for managing data catalogs across various systems, addressing common challenges such as data fragmentation, inconsistent governance, lack of metadata, and the evolving data landscape. By providing a centralized access and governance layer, Gravity simplifies data management, enhances collaboration, and improves scalability, allowing data teams to efficiently handle data assets from diverse sources like cloud data lakes, PostgreSQL, and Snowflake. Key features include universal search, fine-grained access controls with role-based and attribute-based access control (RBAC and ABAC), data product capabilities for attaching business context, and an open architecture that supports integration with multiple data environments. Gravity aims to streamline the discovery, organization, and sharing of data, ultimately enabling organizations to deliver high-quality data assets to stakeholders with less effort.

Starburst Galaxy has introduced enterprise-grade security features, row filtering and column masking, to enhance data protection by restricting access to sensitive information based on user permissions. Row filtering allows data administrators to dynamically filter data tables according to user roles, ensuring that users only see data relevant to their roles, such as sales teams viewing orders from their specific regions. Column masking, set to launch soon, will allow sensitive data like credit card numbers to be partially obscured, maintaining privacy while still allowing access to necessary information. These features are part of Starburst's broader strategy to implement attribute-based access control (ABAC), which will include user-based attributes and fine-grained tagging permissions, enhancing security and flexibility in data management. Users can explore these features through Starburst's free trial or educational resources to better understand role-based access control (RBAC) and ABAC in the platform.

Starburst Galaxy's inaugural launch week, held from June 20th to 23rd, 2023, introduces new features and enhancements aimed at improving data performance and security. Each day of the event highlights a new feature or set of features focused on Starburst Galaxy's core principles of simplicity, access, optionality, and scalability, accompanied by tutorials, videos, and opportunities to win swag. The launch event emphasizes transparency in development, offering insights into the company's roadmap. The week kicks off with the release of row filters and column masks, which enhance enterprise-grade security by providing fine-grained governance and security to cloud data sources. Users are encouraged to visit Starburst's website or follow their social media channels for daily updates on feature releases.

Organizations are increasingly adopting data products to meet the growing demand for data and analytics, yet developing and scaling these products can be challenging due to common pitfalls. These include focusing on quantity over quality, ignoring stakeholder feedback, overlooking team collaboration, and delaying governance. To avoid these traps, it is essential to establish Minimal Viable Data Product (MVDP) standards to ensure quality, engage in iterative development to incorporate stakeholder feedback, foster collaboration across diverse teams with a strong product owner, and implement automated governance to maintain consistency and trust. By addressing these challenges, organizations can deliver high-quality data products that align with stakeholder needs and drive business success.

Trino Fest 2023, a free virtual event hosted by Starburst on June 14-15, invites the Trino community to explore the advancements and expansion of Trino, a SQL query engine designed for data lakes initially with Hive and Hadoop, and now supporting modern data solutions like Iceberg, Delta Lake, and Hudi. The event features presentations highlighting the engine's performance, federated queries, and integration with numerous data sources, showcasing the continued growth in use cases and new community additions. Sponsored by AWS and Alluxio, Trino Fest will include insights from notable companies like Bloomberg, Salesforce, and Stripe, emphasizing its role in the analytics space. The event follows the success of the Trino Summit in San Francisco last November and aims to further enrich the community by presenting new features and integrations that have emerged since 2022.

Data mesh, a concept introduced by Zhamak Dehghani, offers a decentralized approach to managing data, moving away from traditional centralized architectures like data lakes and warehouses. It focuses on distributing data ownership and creating reusable data products tailored to specific needs, adhering to agreed standards and schedules. The data mesh framework is built on four pillars: domain ownership, data as a product, self-service data infrastructure, and federated computational governance. While not suitable for every organization, particularly those lacking sufficient resources or complexity, the principles of treating data as a product can be beneficial across various business contexts. Challenges in implementing data products include adapting to changes in roles and processes, acquiring necessary skills, and managing potential data chaos, which can be mitigated by starting small, utilizing a data product platform, and employing automated governance standards. Emphasizing data as a product ensures that stakeholder needs and value delivery remain priorities, with the potential for organizations to transition to a data mesh approach as they evolve.

Starburst Enterprise is set to deprecate support for legacy Cloudera platforms, including CDH 6.x, HDP 3.1, and Apache Sentry, in its August 2023 Long-Term Support release, with a complete discontinuation scheduled for June 2024. This decision aligns with Cloudera's cessation of support for these platforms in 2022 and the transfer of Apache Sentry to the Apache Attic in 2020. Starburst recognizes the potential challenges for customers and will provide transitional support for twelve months following the deprecation to aid in the migration to alternative solutions. During this period, no new deployments using these end-of-life products will be supported. Customers are encouraged to plan for these changes and seek assistance from their account support teams if needed.

Dan Brault's article explores the strategic importance of choosing the right data lake and analytics architecture for startups, emphasizing the benefits of open file and table formats and distributed query engines. He argues that while cloud data warehouses may initially seem appealing, they often lead to vendor lock-in and scalability issues as businesses grow. Instead, modern data lakes, which integrate open file formats like Parquet and flexible table formats such as Apache Iceberg, allow for scalable, cost-effective data management that preserves business agility and control over data. The article highlights the advantages of using open-source technologies for fostering innovation and flexibility, and it introduces Starburst as an ideal analytics engine built on Trino, designed for startups to execute fast, scalable queries across diverse data sources. By adopting a modern data lake architecture, startups can overcome data access challenges, improve performance, and enhance governance, ultimately unlocking the full potential of their data for informed decision-making.