September 2022 Summaries

Starburst's approach to delivering text search capabilities directly on data lakes addresses the challenges of conducting agile text searches on massive datasets without moving data to proprietary platforms. By utilizing their Smart Indexing and Caching technology, which incorporates the open-source Apache Lucene library, Starburst enables efficient text searches by cutting data into nanoblocks, allowing for optimized indexing at a granular level. This method reduces the cardinality challenge and supports advanced text search applications such as logs analysis, cyber threat detection, and marketing analytics. The solution is designed to be cost-effective, minimizing total cost of ownership and maintenance time, thereby accelerating innovation for data-driven organizations.

Executives are increasingly interested in Data Mesh as a solution to data management challenges, such as delays in data-driven decision-making and issues with data literacy, quality, and legacy technology. According to a PwC report, while 93% of IT managers have discussed Data Mesh within their organizations, only a fraction have a concrete strategy for implementation. The study highlights that Data Mesh offers a decentralized approach that could enhance data analytics quality and reliability, but its adoption depends on the maturity of an organization's data infrastructure. Despite its potential, the transition to Data Mesh involves significant changes in people, processes, and technology, making it a sociotechnical shift.

Starburst, a company built on the open-source Trino project, offers a distributed query engine that supports data lakes, object storage, and data federation, enabling SQL queries across various data sources. Originally stemming from the Hadapt startup, which was acquired by Teradata, Starburst was founded by a team in Warsaw, Poland, who had previously worked on the development of Presto, the precursor to Trino. The company has grown significantly, establishing itself in Poland and securing its first major customer, Zalando, in the EMEA region by addressing their data governance and scalability needs without locking them into specific storage solutions. Starburst's rapid growth and dynamic work environment continue to attract talent as they expand their engineering team and product offerings, emphasizing the importance of adaptability and innovation in the fast-evolving data space.

Data indexing is a crucial strategy for improving query performance and reducing the need for extensive computing resources, but its effectiveness depends on aligning indexes with specific query needs. To benefit from indexing, one must understand business requirements and ensure indexes are designed to avoid full scans, which are resource-intensive. The construction of SQL queries also significantly impacts performance, where efficient join strategies and table ordering can drastically enhance query speed. Index management should adapt to evolving business questions, especially as traditional row-based indexes may not suit columnar big data layouts, necessitating intelligent, dynamic indexing systems. A novel approach using nano-blocks allows for flexible and efficient big data indexing, enabling rapid analytics and accommodating schema changes without the limitations of traditional partitioning methods.

Jordan Zimmerman, a Staff Software Engineer at Starburst, concludes a series on modern Java with a focus on building a JSON serialization library. The series culminates in the creation of a high-level API for the library, which simplifies JSON serialization and deserialization processes, making them more user-friendly. The API includes methods for serializing objects to strings or writers and deserializing JSON strings or readers into objects, leveraging components like JsonPrinter, JsonParser, JsonSerializer, and JsonDeserializer. The series also provides an example using Java 19 to demonstrate the library's functionality, showcasing how a list of custom Card objects can be serialized and deserialized while maintaining data integrity. The post encourages readers to explore further advancements in Java and invites them to try Starburst Galaxy's free trial to enhance data performance, while also extending an invitation to join their team.

In the continuation of the "Bleeding Edge Java" series, the article explores the advanced techniques of deserializing complex Java types, building on the foundational framework discussed in the first part. The process involves updating the implementation of `deserializerFor` to handle a variety of types, including collections, optionals, records, and enumerations, by using specific patterns and deserialization strategies for each type. It introduces the concept of "type tokens" to manage generic types during runtime, despite Java's type erasure, allowing for the deserialization of complex structures like lists of records or optionals. The article culminates in a practical demonstration using `jshell` to serialize a Java record to JSON tokens and then deserialize it back to ensure the integrity of the process, showcasing a complete round-trip conversion between Java objects and JSON. The series invites readers to test the code and highlights the potential of these techniques in real-world applications, also extending an invitation to join the Starburst team.

Following the COVID-19 pandemic, businesses have accelerated their digital transformation efforts, leading to five key big data trends. The adoption of data lakes is on the rise, driven by the need for efficient data manipulation in remote and hybrid work environments, with the market expected to grow significantly by 2026. The convergence of streaming and static data sources is improving predictive capabilities by enabling comprehensive analytics, which will become more prevalent as cloud storage and applications eliminate resource limitations. Data sharing is simplified through cloud migration, broadening the consumer base for data analysis, with cloud providers offering solutions for both internal and external data exchanges. Machine learning is being integrated into databases to optimize queries and dynamically adjust to unprepared data, promoting a separation between data storage and consumption. Finally, predictive analytics is driving the development of next-generation digital applications, shifting from static dashboards to live, dynamic data access, democratizing insights across departments for quicker adaptation and improvement, and combining cloud, database, and analytics advancements for a competitive edge.

Deserialization is a complex process where JSON token streams are converted into Java objects, and it is considered more challenging than serialization due to the complexity of Java objects compared to the simplicity of JSON. The article discusses a design approach that resembles a recursive descent parser, where deserializers are defined for all supported Java types, each responsible for processing required fields. The deserialization process involves creating a TypedDeserializer for each type, accepting JSON tokens, and recursively calling new deserializers when encountering new types, ultimately building the Java object from the processed tokens. Simple types like numbers and strings are easier to handle, while collections and records require more sophisticated deserialization strategies, which will be addressed in a follow-up article. The implementation also introduces a stream Collector mechanism to reduce JSON tokens into a single Java object, and the article promises further exploration into handling complex types in subsequent discussions.

The article explores the process of parsing JSON text in Java, focusing on converting a stream of characters into JSON tokens using a Java interface called JsonParser. It outlines the challenges encountered in this process, such as the need for lookahead when identifying token types and handling character reprocessing for number tokens. The implementation involves using Java's IntStream and iterators, with a specific focus on managing streams of unknown length and converting them into streams of tokens. The article provides a detailed breakdown of the internal workings of the parser, including the handling of whitespace and object names, and demonstrates the parsing process through the use of a Java record serialized into JSON and parsed back into tokens using a series of Java files. The text concludes with an invitation to try Starburst Galaxy's free trial and a note on hiring opportunities for Java enthusiasts.

In this part of the Bleeding Edge Java series, Jordan Zimmerman explains the process of printing JSON text from a stream of JSON tokens using Java. The approach involves defining a Java interface, `JsonPrinter`, which uses a mapping function that can be passed to the JDK Stream's map() method, converting each JSON token into a string representation through an enhanced switch statement and pattern matching. This implementation leverages Java's deconstruction feature and covers all possible cases of the `JsonToken` hierarchy, removing the need for a default case. The article provides code examples to demonstrate how to serialize a Java record into JSON text using a `JsonSerializer` and `JsonPrinter`, with downloadable files for testing in jshell. This development is part of a broader exploration of Java's latest features and is aimed at improving data performance, aligning with Starburst's recruitment efforts for software engineers.

The text explores a serialization framework in Java, specifically focusing on converting Java objects into JSON tokens using Java 19's enhanced features like record patterns and pattern matching for switch statements. It highlights the straightforwardness of serializing simple Java objects such as primitives, strings, and enumerations, while noting the complexities traditionally associated with Java class serialization. With the advent of Java records, extra metadata like annotations is no longer necessary, simplifying the serialization process. The framework restricts serialization to specific Java instances that do not require extra metadata, and the code implements this using a deceptively simple interface, JsonSerializer. The text further delves into the mechanics of serializing collections and records, demonstrating the use of Java's enhanced switch statements and pattern matching to streamline this process. It emphasizes the compactness and simplicity of the framework, which can be run in jshell, and provides a brief overview of setting up and testing the serialization process using Java 19's preview features.

Security data lakes offer a modern solution for organizations aiming to enhance their security analytics capabilities by centralizing and storing vast amounts of data for investigations, threat detection, and compliance. As companies increasingly adopt these systems, challenges such as long-term data retention, instant access, scalability, and cost-efficiency become paramount. Traditional methods often suffer from inefficiencies, with significant resources wasted on scanning and filtering operations. Starburst's innovative approach with big data indexing technology addresses these issues by enabling rapid access to relevant data across multiple dimensions and formats, leveraging indexes like Bitmap and Lucene for text searches. This method not only minimizes computational waste but also supports real-time threat detection and incident management, meeting the critical success criteria for effective security data lake operations.

In the landscape of big data analytics, the choice between micro-partitioning and indexing can significantly impact both performance and cost efficiency. Micro-partitioning divides data into smaller blocks using a predefined subset of columns, allowing for parallel processing but often necessitating reading large data volumes, especially with complex joins. Conversely, traditional indexing separates data storage from access methods, optimizing retrieval and enhancing performance for join-based queries, though it can be cumbersome to design and maintain, particularly with columnar data layouts. Starburst's Smart Indexing and Caching introduces an advanced indexing method utilizing nanoblocks, which are small, dynamically created index sections that boost query performance by minimizing data retrieval needs. This approach, integrated with columnar storage, combines the efficiency of indexing with the simplicity of micro-partitions, adapting dynamically to data changes and query patterns. As demonstrated with ride-sharing data analytics, this method offers substantial performance improvements, emphasizing the need to evaluate query complexity and data structure when choosing the optimal analytics engine.

Teresa Tung, Cloud First Chief Technologist at Accenture, discusses the importance of shifting from a data project mindset to a data product mindset in organizations, emphasizing the benefits of this transition in achieving data democratization and fostering a business-led, product-centric approach. The implementation of Data Mesh principles is highlighted as a strategic move to enhance scalability, flexibility, and data-driven resiliency amid economic uncertainties. Key performance indicators (KPIs) are critical to measuring success, focusing on operational efficiencies, trustworthiness, real-time data capabilities, accessibility, and usability, which collectively enhance user experience and open new business streams. This approach facilitates a continuous release cycle, integrates business and technology outcomes, and encourages collaboration within and beyond the enterprise. The data product mindset transforms how organizations interact with data, moving from isolated projects to reusable, scalable data products that drive business value and innovation.