Apache Iceberg - Open Table Format Specification
Key Insights
- Apache Iceberg defines an open table format for petabyte-scale analytic datasets with ACID transactions, schema evolution, partition evolution, time travel, and snapshot isolation on cloud object storage.
Edit on GitHub — registry.json
Background
Data lakes solved storage cost but shipped with a broken promise: a folder of Parquet files is not a table. No atomicity across files, no schema contract, no query consistency. Apache Iceberg addresses this with an open table format specification — a metadata layer that turns object storage files into first-class database tables without locking data into any engine.
The Metadata Model
Iceberg separates data files from metadata. A table's current state is a single metadata file pointing to a manifest list; each manifest tracks a set of data files with column-level statistics. Writers create new files and atomically swap the metadata pointer (an optimistic-concurrency CAS on the catalog), giving readers snapshot isolation: every reader sees the table at the version it opened.
Deep Dive
That foundation yields the guarantees that made Iceberg the lakehouse standard. ACID transactions via metadata commits rather than file locking. Schema evolution — add, rename, drop, reorder columns — as metadata-only changes, so tables evolve without rewrites. Partition evolution: partitioning rules can change without rewriting existing data, and "hidden partitioning" makes partition predicates invisible to the user because the engine derives them from values. Time travel reads any retained snapshot. File statistics enable scan pruning, and the open specification means Spark, Trino, Flink, DuckDB, and others query the same table through any Iceberg-compatible catalog (REST, Hive, Nessie).
Why It Matters
Iceberg (with Delta Lake and Hudi) is the load-bearing layer of the lakehouse architecture: it provides the warehouse guarantees — ACID, schema, snapshots — on open data. Choosing a table format is now a core data-platform decision, and Iceberg's metadata design is the reference for why.
Key Takeaways
- Catalog choice governs concurrency behavior; use a catalog that implements optimistic locking reliably.
- Hidden partitioning removes the biggest partition-management footgun — stop embedding dates in paths.
- Manage snapshots: expire old ones and compact small files or storage costs and scan times creep.