The unbundling of cloud data warehouses

The unbundling of cloud data warehouses refers to the separation of storage from compute, a trend highlighted by Tanya Bragin in a discussed episode of the "Data Engineering Podcast," which is transforming how analytical workloads are managed. This approach allows data lakes to serve a variety of use cases by leveraging various engines with features such as ACID transactions, SSD caching, indexing, and enterprise-grade security through table formats like Iceberg, Delta Lake, and Hudi, avoiding storage lock-in with a single vendor. Companies are encouraged to initially store their data in a lake/object store, with the option to choose different technologies for specific needs, such as OLAP databases for real-time analytics and high-performance search solutions. As previously closed architectures like Snowflake and BigQuery begin to support external customer storage, the trend toward open analytical environments is becoming more pronounced, despite potential challenges like metadata lock-in. The year 2024 is anticipated to be pivotal for organizations embracing these open architectures, as they can harness the benefits without the constraints of vendor lock-in.