April 2023 Summaries

The partnership between dbt Labs and Starburst offers a streamlined approach to managing data transformations by integrating Starburst as a native connection option in dbt Cloud, allowing data teams to efficiently build, test, and document data pipelines across multiple sources. Starburst, a data lake analytics platform, enhances data discoverability and governance, addressing common challenges such as poor documentation and outdated pipelines. The integration enables data teams to benefit from features like version control and generated documentation, facilitating the creation of production-grade pipelines. Starburst Galaxy's ability to federate queries from dbt Cloud to multiple data sources simplifies the process of building pipelines from a central location, accommodating both relational and NoSQL databases. With this collaboration, users can rapidly set up data transformation projects by creating a dbt Cloud project, selecting Starburst as the data platform, and leveraging SQL to connect diverse data sources efficiently.

Starburst Enterprise platform (SEP) 413-e STS, based on Trino 413, has been released as an upgrade to the 411-e STS version, bringing new enterprise features and enhancements alongside recent Trino updates. A notable addition in this release is the support for Apache Ranger, enabling customers to manage role-based access control for their data products. This allows the creation of access control policies within Ranger Access Manager, which can grant or restrict users' permissions to view, manage, and publish data products in SEP.

In a discussion with Brett Thebault, Co-Founder at Aginic, the focus was on the transformation of data management and analytics within organizations, emphasizing the importance of enablement in achieving successful digital transformations. Aginic, grounded in principles of agility and analytics, advocates for building strong internal teams and empowering them with the right tools and processes, aligning data analytics ambitions with resource capabilities. The conversation highlighted the advantages of decentralized, hybrid architectures in cloud migrations, noting that accessibility has become a crucial metric alongside cost, performance, reliability, and security. A data federation approach is recommended to maintain seamless access to data across different environments, avoiding the pitfalls of a centralized "data cathedral." Thebault stressed the need for agility and flexibility in data products, encouraging a culture of curiosity and self-service among business teams to efficiently leverage data lakes and federated data systems, which drive cost efficiency and enable rapid innovation in data analytics.

Sorted Iceberg tables significantly enhance query performance and reduce cloud storage costs by organizing data according to one or more columns, thus minimizing the number of files read during data retrieval. This sorting approach drastically cuts down on query times, especially in large datasets, by only accessing necessary files rather than scanning all data, as demonstrated with the TPC-DS benchmark. For example, a sorted version of the catalog_sales table on the cs_sold_date_sk column showed a substantial decrease in data read compared to its unsorted counterpart. Implementing these sorted tables in Apache Iceberg, particularly in conjunction with the Starburst Galaxy platform, provides an efficient solution for optimizing data storage and retrieval processes. Additionally, materialized views can also benefit from sorted columns, further enhancing performance. Iceberg's optimize command consolidates smaller files into larger, sorted ones, maintaining performance advantages even as data is streamed or batch processed. This methodology not only improves performance but also yields cost savings in cloud object storage, making it a valuable strategy in data management.

At the Datanova conference, Bank of America leaders Amy Avery and Michelle Boston discussed the bank's strategy for integrating data management with analytics to enhance customer experiences and drive growth while maintaining data privacy and security. They emphasized the importance of trust and collaboration between data and analytics teams, highlighting how intertwined roadmaps and clear roles facilitate effective data use. The bank's approach prioritizes understanding the "why" behind customer needs and business objectives, allowing cross-functional teams to work cohesively. This philosophy aligns with a broader industry shift towards measuring success through value creation, rather than just data protection. Technologies like those from Starburst and ThoughtSpot are instrumental in this strategy, enabling real-time analytics and business intelligence without compromising data management standards, thus helping institutions like Bank of America focus on delivering aligned business outcomes.

Starburst Galaxy offers a flexible solution for managing cloud data warehouse (CDW) workloads by enabling data to be stored in open formats like Parquet and ORC within a user's cloud account, thus avoiding vendor lock-in and high costs associated with traditional CDWs. The platform facilitates real-time data analysis across diverse sources, including object storage, relational, and non-relational systems, using a wide range of connectors and a SQL cost-based optimizer, allowing companies to perform queries without needing to copy data into a CDW. Built on the open-source Trino engine, Starburst Galaxy supports multiple concurrent users and provides the ability to choose different engines for analytics tasks, enhancing the adaptability and scalability of data processes. Additionally, it is designed to transform data swamps into well-structured, high-performing data lakes, offering total data ownership and the flexibility to integrate other engines as needed.

In an episode of Data Mesh TV, Adrian Estala and Nachiket Mehta discuss the complexities and strategies of implementing a persona-driven data mesh design. They emphasize the importance of considering all personas—data producers, consumers, and engineers—in the early design phase to ensure enterprise-wide commitment and alignment. The mesh approach offers transparency for data producers, flexibility and fit-for-purpose design for data consumers, and potential relief for overwhelmed data engineers by enabling autonomous domain teams. A key highlight is the development of domain-driven ownership through data ontology, which promotes best practices and addresses technical and ethical concerns in data management. The episode also touches on the concept of a shared data economy and the challenges of establishing clear boundaries for data sharing, alongside the evolving role of data contracts and service level agreements in governance and operational efficiency.

Trino Fest 2023, hosted by Starburst, is a virtual event scheduled for June 14th and 15th, celebrating the Trino data lakehouse query engine. The event aims to foster community engagement among Trino enthusiasts and data professionals, offering opportunities for learning, networking, and sharing experiences related to data lakehouse management. The focus will be on Trino's role in data lake and lakehouse architectures, including discussions on Apache Iceberg, Delta Lake, and Hudi, as well as topics like query federation, data governance, and SQL processing with Trino. A call for speakers has been issued, inviting submissions on relevant use cases and experiences, with the deadline set for May 19th. Sponsorship opportunities are available, offering benefits such as logo placement and social media promotion. Registration is open for those interested in gaining insights and tips for managing data lakehouses.

Starburst Enterprise Platform (SEP) 411-e STS has been launched following the 410-e LTS release and is based on the Trino 411 release, bringing several new features and enhancements. Key improvements include the migration procedure to convert Hive tables to Iceberg, join and LIKE pushdown in Ignite, support for DELETE in Ignite, and a procedure table function for executing stored procedures in SQL Server. The release also offers faster join queries over Hive bucketed tables and improved planning for tables with many columns in Hive. SEP 411-e enhances performance with managed statistics for Redshift and SQL Server connectors, introduces a preview-sharing queries tab and query editor improvements, and provides numerous security advancements for OAuth 2.0, built-in access control, and Lake Formation support.

Starburst Galaxy has introduced attribute-based access control (ABAC) to enhance governance and security across cloud data sources, providing a more dynamic and flexible access control framework compared to traditional role-based access control (RBAC). ABAC allows organizations to define access rules based on a diverse set of attributes, including user roles, job titles, and contextual data, thus facilitating precise and real-time policy enforcement that aligns with business-driven data privacy policies. This approach supports the principle of least privilege (PoLP) by granting users only the necessary permissions for their tasks, thereby minimizing security risks. To implement ABAC in Starburst Galaxy, users must first tag their data assets, which involves creating a hierarchical tagging system that applies attributes to various data levels like catalogs, schemas, tables, and columns. These tags are then used to create policies that specify access privileges, potentially with expiration dates, to manage data access dynamically across complex enterprise environments.

Starburst Galaxy's April 2023 update introduces enhanced security and governance features aimed at simplifying data management throughout its lifecycle. The platform now supports attribute-based access control (ABAC) in public preview, allowing administrators to create scalable security policies using business context rather than specific data entity names, which addresses issues with the rigidity of role-based access control (RBAC). This update also includes the ability to assign tags, such as those identifying personally identifiable information, to manage access more effectively. Additionally, the much-anticipated AWS PrivateLink support is available in private preview, offering an extra layer of security and network reliability by enabling encrypted connections between data sources and Starburst Galaxy. Looking ahead, the platform plans to introduce further security features, such as row-level filtering, single sign-on (SSO) for Tableau and PowerBI, and security APIs to manage policies programmatically.

SEMMA and CRISP-DM are two prominent process models used in data mining and machine learning to develop predictive models and extract insights from data, each with unique characteristics and applications. CRISP-DM, developed in the late 1990s, is a comprehensive and widely recognized framework encompassing six phases: business understanding, data understanding, data preparation, modeling, evaluation, and deployment, making it flexible and suitable for various projects. In contrast, SEMMA, developed by SAS, focuses on five phases—sample, explore, modify, model, and assess—primarily targeting the modeling phase and is closely linked to SAS software, often used alongside broader methodologies like CRISP-DM. While CRISP-DM is widely adopted due to its extensive documentation and community support, SEMMA is more specialized and specific to the SAS environment, making CRISP-DM generally the more practical choice for diverse data mining projects.