March 2023 Summaries

During a fireside chat at the Datanova conference, Starburst Co-Founder and CEO Justin Borgman, alongside industry leaders from Bank of America, Wells Fargo, and Accenture, discussed the evolving landscape of financial services and data strategies. The conversation emphasized the growing importance of distributed data architectures, moving away from centralized data storage solutions that often fail to deliver on their promises. Instead, organizations are encouraged to adopt data mesh concepts that decentralize data ownership, allowing data owners to take responsibility for governance and quality, thereby enhancing data utilization and insight generation. Moreover, the discussion highlighted the need for empowering employees with self-service analytics capabilities to drive productivity and innovation, while also addressing operational and regulatory risks. The panelists noted that modernizing data infrastructure isn't solely about cost reduction but also about enabling market growth and compliance. Ultimately, a cultural shift towards a data-first mindset is essential for capitalizing on the business impact of distributed data, with Starburst aiding financial institutions in accelerating insights and reducing costs through enhanced data strategies.

Starburst has announced new integrations with Google Cloud's Dataplex and BigQuery, expanding its capabilities for handling distributed data across various cloud and on-premise systems. These integrations aim to help organizations manage and analyze their data more effectively by allowing Starburst to function as a distributed query engine within Google's data solutions. This enables the discovery and governance of federated data products, even from sources outside Google Cloud, facilitating the creation of a unified data mesh. Starburst enhances BigQuery's analytics by enabling it to read datasets from diverse locations, broadening the accessibility of Google Cloud's tools and accelerating business insight discovery. The new features are available for trial through BigQuery Partner Center, offering organizations a chance to experiment with these capabilities and extend their data reach.

Starburst Warp Speed enhances Trino's capability to perform large-scale geospatial analyses by significantly improving the cost and performance efficiency of queries through smart indexing and caching. The technology automatically accelerates queries, enabling users to achieve up to a 6x improvement in speed, as demonstrated in experiments involving datasets with billions of rows. This improvement addresses common challenges faced by Trino users in handling computationally intensive geospatial operations, such as spatial joins and nearest neighbor searches, without the need for complex configurations. By leveraging Warp Speed, users can conduct extensive geospatial analyses while maintaining a balance between performance and business needs, overcoming issues related to data partitioning and index creation.

Trino, an open-source distributed query engine, is adept at running geospatial queries across various data sources without needing extensive data modeling, making it ideal for tasks like ad-hoc analysis. It is OpenGIS compliant and supports a broad range of geospatial functions, facilitating the unification and joining of geospatial data from multiple sources. The text outlines two methods for optimizing geospatial queries using Trino: leveraging its native geospatial support and utilizing Bing tiles for more efficient data segmentation and retrieval. An example demonstrates the use of Trino’s Hive connector to run queries on large datasets, like a public ride-sharing dataset, showcasing techniques such as bounding boxes and Bing tile segmentation to enhance query performance and reduce data read times. By applying these methods, significant improvements in query response times are achieved, highlighting Trino's capability to efficiently handle geospatial workloads on data lakes.

Richard Teachout, CTO of El Toro, emphasizes the importance of fostering curiosity in data engineering to drive innovation and achieve significant results. Drawing from his 30 years of experience, Teachout identifies key strategies such as making exploration accessible through interactive visualizations, promoting continuous learning, embracing failure as a learning opportunity, providing advanced tools, and encouraging creative activities outside the workplace. He highlights how curiosity has led to substantial improvements at El Toro, including significant reductions in server usage and the development of new tools like a natural-language SQL interface. By nurturing an environment that balances exploration with accountability, El Toro has transformed from an advertising firm into a successful analytics company with patented technologies, exemplifying the power of curiosity-driven progress.

Data lakes are being increasingly recognized for their potential to serve as comprehensive solutions for storing and analyzing data, rivaling traditional data warehouses. Companies are beginning to appreciate the structured capabilities of data lakes, which can support various analytical use cases, including multi-terabyte queries and real-time analytics, while offering enhanced security and flexibility. Innovations such as Apache Iceberg and Delta Lake enable data modification and compliance with regulations like GDPR. A modern data lakehouse architecture, particularly when built on open-source, vendor-agnostic formats, can future-proof data strategies by avoiding vendor lock-in and reducing costs. As the market evolves, more companies are opting to consolidate their analytics within a single data lake environment, driven by its simplicity and cost-effectiveness, and the ability to leverage multi-engine approaches that provide flexibility and ease of access for data teams.

The Starburst Enterprise 407-e LTS release introduces enhanced capabilities including the highly anticipated Warp Speed feature, which improves data lake query performance by adding an indexing and caching layer to object storage catalogs, boosting performance up to seven times while reducing cloud compute costs by up to 40%. This release also debuts managed statistics, enabling smarter query optimizations across multiple data sources, and enhances security with expanded access control features and integrations with AWS Lake Formation. Connectivity improvements include expanded fault-tolerant execution and a new Cosmos DB connector, facilitating more advanced analytics and data transformations. The user experience is further refined by a new saved queries pane and enhanced troubleshooting tools, underscoring Starburst's commitment to customer support. These advancements aim to optimize data lake analytics, providing faster, cost-effective solutions without necessitating data movement or modeling changes.

Data products play a crucial role in transforming how organizations leverage data, aligning closely with the concept of a data mesh, which focuses on decentralized data management to maximize business value. The lifecycle of a data product starts with the data product manager aligning these products with business goals, followed by a series of stages that include building, maintaining, and retiring data products. Unlike traditional data management approaches, data products require versioning, sharing, and applying product management disciplines to ensure their relevance and utility. A data product combines deep data knowledge with domain expertise and is defined by attributes like discoverability, security, and interoperability. The integration of a metadata plane and reusable frameworks enhances the management and scalability of data products, while DataOps practices ensure automation and observability throughout the process. Financial services and other regulatory industries are particularly poised to benefit from data products due to their complex data needs, and partnerships like that of Starburst and Deloitte are helping organizations derive greater value from their data investments by utilizing data products.