Apache Iceberg is an open table format for managing large-scale analytic datasets on object storage (S3, ADLS, GCS). Originally developed at Netflix to address Hive limitations, Iceberg brings ACID transactions, schema evolution, and time travel to data lakes, enabling reliable lakehouse architectures. Unlike file formats (Parquet, Avro, ORC), Iceberg defines how data files are organized into logical tables with consistent point-in-time snapshots. The key insight: Iceberg replaces expensive directory listings with a three-layer metadata tree (metadata JSON → manifest lists → manifest files → data files), enabling massive scalability—production deployments manage petabyte-scale tables with tens of millions of files. What distinguishes Iceberg is hidden partitioning (users query raw values, transforms happen transparently), partition evolution (change partitioning without rewriting data), and vendor-neutral governance under the Apache Software Foundation with the broadest multi-engine support across Spark, Flink, Trino, Snowflake, BigQuery, and DuckDB. Apache Iceberg 1.11.0 (released May 2026) marked the GA maturity of format version 3 (V3), delivering deletion vectors, the variant type, nanosecond timestamps, geospatial types, row lineage, and the pluggable File Format API.
What This Cheat Sheet Covers
This topic spans 23 focused tables and 201 indexed concepts, 127 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
A jump-to index of every table row in this cheat sheet.
An interactive map of every table and concept in this topic.
Table 1: Core Table Format Concepts
Iceberg's metadata-driven architecture is its defining advantage over directory-based formats like Hive. Understanding the three-layer metadata tree—and how atomic commits, snapshots, and hidden partitioning interact—is the foundation for everything else.
| Concept | Example | Description | |
|---|---|---|---|
Defines schema, partitioning, snapshots for data files | • Open specification for organizing raw data files (Parquet/Avro/ORC) into logical tables with ACID semantics • separates metadata from data storage | ||
metadata/v1.metadata.json | Three-layer architecture: metadata JSON (schema, snapshots) → manifest lists (snapshot metadata) → manifest files (file statistics) → data files | ||
snapshot_id=8744736658442914487 | • Immutable point-in-time view of table • created on every commit • enables time travel and rollback | ||
snap-8744736658442914487-1-abc123.avro | • Avro file containing references to all manifest files for a snapshot • tracks partition ranges and file counts per manifest | ||
abc123-m0.avro | Avro file tracking subset of data files with per-file statistics (path, partition, record count, min/max, null counts) | ||
data/order_date_month=2024-03/0001.parquet | • Physical file containing table data in Parquet, Avro, or ORC format • immutable once written | ||
REST, Hive Metastore, AWS Glue, Nessie | • Metadata store that tracks table locations and current metadata pointer • provides atomic updates and multi-table transactions | ||
ALTER TABLE ADD COLUMN age INT | • Add, drop, rename, reorder columns without breaking queries • uses column IDs instead of positions • no "zombie data" on column add | ||
Change month(order_date) → day(order_date) | • Modify partitioning strategy without rewriting historical data • old and new partition specs coexist in same table | ||
SELECT * FROM table FOR SYSTEM_TIME AS OF '2026-01-15' | • Query table as it appeared at specific timestamp or snapshot ID • access historical data without backups |