Apache Iceberg Architecture Explained

Apache Iceberg is an open table format for data lakehouses that brings ACID transactions, schema evolution, time travel, and concurrent multi-engine access to object storage without relying on directory-based table layouts. Its architecture consists of a catalog that atomically points to current table metadata, a metadata hierarchy of JSON metadata files and Avro manifests that records snapshots, schemas, partitions, and file statistics, and immutable Parquet or ORC data files with optional delete files. Query engines use manifest-level partition summaries and file-level statistics to prune unnecessary files, while writes create new files and commit through an optimistic-concurrency catalog pointer swap, providing snapshot isolation. Snapshots support time travel and rollback, while mutable branches enable isolated development and immutable tags preserve audit or compliance states. Iceberg v2 added row-level deletes, and v3 adds deletion vectors, semi-structured VARIANT support, finer timestamp precision, default values, and row lineage, though upgrades require compatible engines. Because immutable writes can accumulate small files, metadata, delete files, and orphaned storage objects, regular compaction, snapshot expiration, orphan-file removal, and manifest optimization are required to sustain performance and control costs.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.