January 2024 Summaries

Starburst, an AWS Specialization Partner and Marketplace Seller, is enhancing organizational data insights by offering data products that integrate with AWS analytics services and data lakes. These products enable real-time query access to disparate datasets, challenging the traditional single-source-of-truth model by facilitating federated access through over 50 connectors. Starburst Data Products simplify the creation, management, and sharing of data, leveraging AWS infrastructure for enhanced performance and cost efficiency. By decentralizing data management and providing centralized visibility, Starburst empowers subject-matter experts to curate and maintain data, which promotes dynamic and reusable data products aligned with core software engineering principles. This approach ensures fine-grained access control and governance, allowing data consumers to extract insights without needing extensive technical skills. The use of Starburst's Stargate connector further enhances data accessibility by enabling seamless data movement across regions, optimizing AWS infrastructure elasticity and reducing compute costs.

The comprehensive Security Data Platform architecture, as detailed by Kadd Systems, offers a robust framework for organizations aiming to build or enhance their security stacks for various use cases such as Security Operations Center (SOC) applications. This platform leverages modern data lakes and warehouses, utilizing tools like Snowflake, Databricks, and Starburst, which is built on open-source Trino, to facilitate a federated approach ideal for hybrid environments. Essential components include data ingestion, where existing pipelines can incorporate security telemetry via tools like Vector.dev and Python-based systems orchestrated with Airflow, and data standardization and normalization, ensuring a uniform data model across multiple sources. The architecture also emphasizes streaming capabilities with platforms like Kafka, serving as a buffer and enabling layered detections, while storage engines can be any major cloud provider or on-prem solutions like MinIO, with Apache Iceberg/Delta serving as the table format. Starburst's compatibility with various specialized platforms such as Neo4j, Elastic Search, and Clickhouse, along with its integration with Jupyter notebooks, enhances capabilities for machine learning and threat hunting. The platform supports a wide array of organizational teams, including SOC, Insider Threat Detection, and Machine Learning teams, offering use cases from on-demand reporting and data correlation to complex threat hunts and integration with collaboration tools like Slack. Its decentralized methodology is particularly advantageous for federal agencies and organizations with hybrid cloud architectures, providing a cost-effective and flexible solution to centralized or decentralized data management.

Data lakehouses have emerged as a hybrid solution combining the scalability of data lakes with the high-performance query capabilities of data warehouses, offering a cost-effective method for storing and analyzing large datasets. While traditional data warehouses require complex and expensive upfront data cleaning and schema declaration, data lakes allow for a more flexible "store-first, organize-later" approach, albeit with limited querying capabilities. Data lakehouses address these limitations by retaining data in read-optimized formats in the data lake and managing schema and metadata through specialized software, similar to data warehouses. However, the inability to query external data systems limits their potential. Data virtualization technology is poised to become integral to data lakehouses, enabling them to query data across an organization's various systems, including traditional data warehouses, thus offering a unified interface for comprehensive data analysis. Advances in networking and machine learning are enhancing data virtualization capabilities, which are explored in a new book discussing the technical challenges and opportunities within this evolving landscape.

Starburst Galaxy's 2023 Wrapped highlights a year of significant product innovations, partnerships, and transformative customer experiences, with advancements in AI features, smart indexing, cross-cloud analytics, and enhanced data observability. Key developments include universal search capabilities, automatic schema discovery, AI-powered data classification, and attribute-based access controls, all designed to improve data management and accessibility. The introduction of Python Data Frames, Warp Speed indexing, and streaming ingest has further enhanced performance and reduced costs for users. Success stories from clients like Halliburton, 7bridges, and Vectra illustrate the impact of these innovations, including improved decision-making, expedited development cycles, and expansion into new markets. Starburst's events, such as Datanova and Trino Summit, have fostered industry enthusiasm and engagement, underscoring the significance of open data lake platforms. Looking ahead to 2024, Starburst plans to continue enhancing user experiences through AI, real-time analytics, and expanded data sharing, while hosting the first in-person Datanova event in New York City.

Starburst has announced the deprecation of RubiX, Alluxio, and Sentry integrations with its Enterprise Platform Hive connector due to maintenance and security concerns, effective immediately. RubiX has been inactive for two years and lacks support from the Trino project, while Alluxio's support was discontinued in October 2019, leading to the decision to enhance security by removing these dependencies. Starburst plans to release an updated Hive connector that excludes RubiX, Alluxio, and Apache Sentry (deprecated in June 2023) to improve security. A legacy version of the Hive connector will be available until June 1, 2024, for users who still rely on these solutions, although it will not include defect or security vulnerability support. For users seeking alternatives, Starburst suggests exploring solutions like Starburst Warp Speed and Starburst Galaxy to enhance data performance.

Starburst Enterprise Platform (SEP) is transitioning to Java 21, aligning with Trino's upgrade strategy to leverage the improvements offered by the latest Java long-term support version. By the February 2024 LTS release, SEP will support both Java 17 and Java 21, allowing users flexibility during the transition. However, by the May/June 2024 LTS release, Java 21 will become mandatory for SEP, following extensive testing and validation to ensure robust support across various deployment mechanisms such as CFT/AMI, Kubernetes, and Docker Images. The upgrade to Java 21 aims to enhance performance and code optimization in Trino, particularly through Project Hummingbird, and SEP encourages users to test Java 21 with existing deployment methods like Starburst Admin and RPM to ensure a smooth transition. For any concerns, users are advised to contact their Technical Account Manager.

Starburst Galaxy has introduced two new features, Jobs and SQL routines, to enhance SQL workflow efficiency and reduce operational overhead for data professionals dealing with complex, repetitive tasks. The Jobs feature allows users to automate and schedule SQL query execution, providing monitoring and logging capabilities for tracking progress and identifying anomalies. SQL routines, limited to scalar user-defined functions, enable users to create reusable custom functions within their queries, enhancing code modularity and simplifying complex operations. These enhancements aim to improve productivity and allow data engineers to focus on strategic tasks, with Jobs in public preview and SQL routines generally available, accessible through a free Starburst Galaxy account.

Starburst has announced that while Trino Community Connectors will no longer be included in the core distribution of the Starburst Enterprise Platform after the 2024 Q1 Long Term Support release, they will remain accessible and usable through the Trino community's Maven repository. Although these connectors, such as Accumulo, Cassandra, and Elasticsearch, are created by the Trino Community and not formally supported by Starburst, customers can still request formal support through their Technical Account Manager or Solution Architect. The company emphasizes its commitment to security and reliability, explaining that certain community plugins will continue to be included in the platform. Starburst has a process in place for promoting Community Connectors to fully supported status, which involves thorough testing by their engineering team and collaboration with various internal departments.

Annalect's adoption of a data mesh framework revolutionized their data management by embracing a decentralized approach, allowing for scalable data governance without the need for a complete architectural overhaul. This method involves categorizing data into independent domains, each managed by smaller, dedicated teams, thus enhancing flexibility and accountability. The implementation of distributed compute tools such as Trino, Starburst, and Amazon Athena enabled seamless data access and integration without centralizing data storage, while data clean rooms facilitated secure external collaborations without moving sensitive data. This innovative approach has not only mitigated the need for frequent, costly re-architectures but also supported the integration of new technologies like large language models (LLMs) and retrieval-augmented generation (RAG), positioning Annalect to meet evolving data demands efficiently.

Starburst Enterprise Platform (SEP) is set to transition to Java 21 in 2024, with the Trino SQL query engine running exclusively on Java 21 as of release 436. While SEP will support both Java 17 and Java 21 through the February 2024 Long Term Support (LTS) release, Java 21 will become mandatory by the May 2024 LTS release. This transition aims to harness the performance improvements and optimizations offered by Java 21, aligning with Trino’s strategy of adopting new Java LTS versions. To facilitate a smooth transition, SEP will support dual Java versions for various deployment mechanisms, encouraging users to test Java 21 in advance. The company has conducted extensive testing to assure robust support, and recommends the Eclipse Temurin OpenJDK distribution from Adoptium for users.

The 2024 Data and AI Leadership Executive Survey highlights the transformative impact of Generative AI on organizational roles, particularly for Chief Data Officers (CDOs) and AI leaders, while also emphasizing the challenges posed by data quality, talent acquisition, and ethical governance. As Generative AI continues to gain traction, its integration remains nascent, largely influenced by the upcoming European Union AI Act, which aims to regulate AI through risk categorization and enforce compliance with ethical and transparent AI practices. The survey underscores the essential role of CDOs and Chief Data & Analytics Officers (CDAOs) in driving data and AI innovation, although these positions face hurdles in becoming fully established. Varied industry-specific trends reflect differing focuses, with less regulated sectors prioritizing growth, while regulated ones emphasize compliance. The survey also identifies cultural shifts and business integration as principal challenges, necessitating a strategic approach to align with regulatory frameworks and maximize AI's potential. The proposed AI Act sets extensive guidelines for high-risk AI systems, emphasizing data ethics and governance, and outlines significant penalties for non-compliance, highlighting the importance of responsible AI implementation.