Apache Iceberg 101: The Table Format Reshaping Data Lakes

Iceberg, initially developed by Netflix and now part of Apache, is an open-source table format designed to address high-scale data challenges by separating storage from compute, allowing reliable data access across multiple concurrent readers and writers. It contrasts with traditional file formats like CSV and Parquet by representing a database table with a cluster of files for data and metadata, often written in Parquet. Iceberg's design supports scalability, cost optimization, mixed-compute support, and open format preference, enabling users to avoid vendor lock-in and integrate with various tools. Its key feature is the separation of storage and compute, allowing for customized storage choices and dynamic compute allocation, which supports data sharing and a Zero Copy/Zero ETL approach. While Iceberg offers numerous advantages, its catalog, which acts as an authority for updates and ensures consistent metadata, poses challenges due to the current limited support across different platforms. Despite these limitations, the ecosystem is progressing towards broader support, and Iceberg's potential benefits make it an attractive option for companies dealing with large-scale data needs.