July 2019 Summaries

Achieving big data "Nirvana" in the cloud involves a structured approach through four stages designed to optimize data management and analytics. Initially, companies face the challenge of disparate data silos across various systems, which complicates data access and increases dependency on IT for resource-intensive ETL projects. Introducing a "consumption layer" with a tool like Trino allows analysts to directly query data sources without knowing their location, thus bypassing IT bottlenecks and enhancing performance. This separation of compute and storage, facilitated by Trino, enables businesses to strategically move to cloud object storage using open data formats like ORC and Parquet, avoiding vendor lock-in. Ultimately, this transformation allows companies to efficiently scale storage and compute resources on demand, offering analysts seamless access to data while optimizing IT expenditures, thus reaching the ideal state of big data management in the cloud.

A consumption layer is an integral part of modern enterprise data architectures, acting as an intermediary between data users and data sources to streamline data accessibility and performance. It allows analysts to execute SQL queries without concern for the data's location or format, effectively insulating them from disruptions caused by data migrations or IT infrastructure changes. This layer offers significant benefits, including mitigating the risks of data movement, simplifying the complexity of maintaining a solutions stack, and preventing vendor lock-in by allowing organizations to connect to various data sources. Starburst Enterprise exemplifies a consumption layer by enabling fast, seamless access to data regardless of its source, thereby enhancing user productivity and IT flexibility during cloud migrations.

Enterprise data architectures have evolved significantly over the decades, transitioning from relational database management systems in the 1980s to cloud-based models in the 2020s. This evolution reflects a necessity to handle increasing data volumes and the associated costs effectively. Historically, each new technological advancement, from enterprise data warehouses to open-source databases and Hadoop, has addressed specific challenges but also introduced complexities and vendor lock-in issues. The current focus is on creating flexibility, termed "optionality," in data architectures by embracing the separation of storage and compute, utilizing open data formats like ORCFile, Parquet, and Avro, and implementing abstraction layers such as query federation and data virtualization. These strategies aim to future-proof data infrastructures by minimizing disruptions during migrations and allowing analysts seamless data access across diverse platforms, thereby decoupling user experience from data location and enabling ongoing adaptation to new technologies without extensive overhauls.