Apache Iceberg Open Table Format Cheat Sheet

Updated 2026-05-15

🧠Study flashcards on this topic81 cards · spaced repetition→

Apache Iceberg is an open-source table format designed for large analytic datasets on cloud object storage (S3, ADLS, GCS), developed originally at Netflix and now an Apache Top-Level Project. It brings ACID transactions, snapshot isolation, schema evolution, and time travel to data lakes by layering a metadata-driven transactional model over immutable data files. Iceberg decouples the physical layout (Parquet/ORC/Avro files) from the logical table structure, enabling features like hidden partitioning, partition evolution, and multi-engine interoperability (Spark, Flink, Trino, Snowflake, BigQuery, Hive, Presto, Dremio, Athena, EMR). One key architectural principle: every write creates a new snapshot — an immutable point-in-time view of the table captured in a manifest list, enabling time travel, versioning, and zero-downtime concurrent writes.

What This Cheat Sheet Covers

This topic spans 16 focused tables and 102 indexed concepts, 81 flashcards. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.

Table 1: Core Concepts and ArchitectureTable 2: Hidden Partitioning and Partition TransformsTable 3: Schema EvolutionTable 4: Time Travel and SnapshotsTable 5: Row-Level Operations and DeletesTable 6: Branching, Tagging, and WAPTable 7: Catalogs and Metadata ManagementTable 8: Compaction and MaintenanceTable 9: Format Versions and Spec EvolutionTable 10: Query Optimization and PerformanceTable 11: Engine IntegrationsTable 12: Write Modes and DistributionTable 13: Stored Procedures and ActionsTable 14: Python and Ecosystem LibrariesTable 15: Metadata Tables and IntrospectionTable 16: Puffin Statistics and Advanced Metadata

Quick IndexSubscribe to unlock

A jump-to index of every table row in this cheat sheet.

Mind MapSubscribe to unlock

An interactive map of every table and concept in this topic.

Table 1: Core Concepts and Architecture

To understand Iceberg you have to understand its layered metadata, because that layer is what turns a pile of Parquet files in object storage into a real ACID table. These terms — snapshot, manifest list, manifest, metadata file, and catalog — name each rung of that hierarchy, and the optimistic-concurrency model that lets writers and readers work without ever blocking each other falls out of it.

Concept	Example	Description
Table Format	Iceberg is a format spec defining how to organize data files, metadata files, manifest lists, manifests, and snapshots into a logical table	• Not a storage engine or query engine, but a metadata layer that sits on top of Parquet/ORC/Avro files stored in object storage • provides table semantics, schema, partition layout, and consistent snapshots
Snapshot	Each write creates snapshot `5237498123985123` with manifest list `s3://bucket/snap-5237498123985123.avro`	• Immutable point-in-time view of a table • captures the state of all data files at commit time • every transaction produces a new snapshot, enabling time travel, rollback, and ACID guarantees
Manifest List	Avro file `snap-123.avro` references manifests: `manifest-1.avro`, `manifest-2.avro`, ...	• Top-level metadata file for a snapshot • lists all manifest files and partition-level statistics (record count, file count, bounds) • enables partition pruning at planning time without opening manifest files
Manifest File	`manifest-1.avro` tracks 50 data files: `data-001.parquet`, `data-002.parquet`, ... with column stats (min, max, null count, NDV estimates)	• Tracks individual data files and their file-level statistics (bounds, null counts, row counts) • reused across snapshots to avoid rewriting unchanged metadata • critical for predicate pushdown and file pruning
Metadata File	JSON file `v3.metadata.json` with schema, partition spec, sort order, snapshots history, current snapshot pointer	• Central metadata file pointing to the current snapshot and containing table schema, partition spec evolution history, and snapshot log • atomic pointer update ensures ACID commits
Catalog	REST, Hive Metastore, AWS Glue, Nessie, JDBC, Hadoop, Polaris — each tracks table location `s3://bucket/db/table` → `v3.metadata.json`	• External service or file system that maps `database.table` names to their current metadata file location • supports atomic compare-and-swap for concurrency control
ACID Transactions	Reader sees snapshot 100; writer commits snapshot 101 using optimistic concurrency — reader still queries snapshot 100 until explicitly moving to 101	• Iceberg uses optimistic concurrency control (OCC) with snapshot isolation or serializable isolation • writers validate parent snapshot at commit time • no locks, readers never block writers

Table 2: Hidden Partitioning and Partition Transforms

One of Iceberg's signature features is that it derives partition values from your columns automatically — you write a partition spec once, and queries filter on the plain source column while Iceberg quietly applies the partition filter. These transforms are the toolkit for that: identity and bucket for categorical and high-cardinality data, truncate for prefixes, and the time transforms (year through hour) for the date binning that drives most analytic tables. Partition evolution closes the set out, letting you change strategy later without rewriting a byte of old data.

Transform	Example	Description
Identity	`PARTITION BY (region)` → partitions use exact values: `region=us-east`, `region=eu-west`	• Uses column value directly as partition value • equivalent to traditional Hive-style partitioning • useful for low-cardinality categorical columns

The rest of this section is part of the full cheat sheet

Table 3: Schema Evolution

Changing a table's shape in Iceberg is almost always a metadata-only edit — no data rewrite, no downtime. The trick that makes this safe is column ID tracking: every column carries a permanent integer ID, so adds, drops, renames, reorders, and type widenings all just update the ID-to-name mapping rather than touching the underlying files, which is exactly where older name-based formats like Hive get themselves into trouble.

Operation	Example	Description
Add Column	`ALTER TABLE orders ADD COLUMN promo_code STRING` — readers see `NULL` for old data; new data includes the column	• Adds column to schema • metadata-only operation (no data rewrite) • old files implicitly return `NULL` for the new column • fully compatible with existing data

The rest of this section is part of the full cheat sheet

Table 4: Time Travel and Snapshots

Because every write produces a new immutable snapshot, you can query the table as of any past timestamp or snapshot ID, read just the rows that changed between two snapshots, or roll the whole table back when a write goes wrong. The catch worth remembering: time travel only reaches as far back as your snapshot retention allows, so expiration policy and your audit window are two sides of the same coin.

Feature	Example	Description
Time Travel by Timestamp	`SELECT * FROM orders FOR SYSTEM_TIME AS OF '2024-05-10 14:30:00'` — reads table state at that timestamp	• Queries table as it existed at a specific timestamp • Iceberg finds the nearest snapshot before or equal to the timestamp • useful for auditing, debugging, and regulatory compliance

The rest of this section is part of the full cheat sheet

Table 1: Core Concepts and Architecture

Concept	Example	Description
Table Format	Iceberg is a format spec defining how to organize data files, metadata files, manifest lists, manifests, and snapshots into a logical table	• Not a storage engine or query engine, but a metadata layer that sits on top of Parquet/ORC/Avro files stored in object storage • provides table semantics, schema, partition layout, and consistent snapshots
Snapshot	Each write creates snapshot `5237498123985123` with manifest list `s3://bucket/snap-5237498123985123.avro`	• Immutable point-in-time view of a table • captures the state of all data files at commit time • every transaction produces a new snapshot, enabling time travel, rollback, and ACID guarantees
Manifest List	Avro file `snap-123.avro` references manifests: `manifest-1.avro`, `manifest-2.avro`, ...	• Top-level metadata file for a snapshot • lists all manifest files and partition-level statistics (record count, file count, bounds) • enables partition pruning at planning time without opening manifest files
Manifest File	`manifest-1.avro` tracks 50 data files: `data-001.parquet`, `data-002.parquet`, ... with column stats (min, max, null count, NDV estimates)	• Tracks individual data files and their file-level statistics (bounds, null counts, row counts) • reused across snapshots to avoid rewriting unchanged metadata • critical for predicate pushdown and file pruning
Metadata File	JSON file `v3.metadata.json` with schema, partition spec, sort order, snapshots history, current snapshot pointer	• Central metadata file pointing to the current snapshot and containing table schema, partition spec evolution history, and snapshot log • atomic pointer update ensures ACID commits
Catalog	REST, Hive Metastore, AWS Glue, Nessie, JDBC, Hadoop, Polaris — each tracks table location `s3://bucket/db/table` → `v3.metadata.json`	• External service or file system that maps `database.table` names to their current metadata file location • supports atomic compare-and-swap for concurrency control
ACID Transactions	Reader sees snapshot 100; writer commits snapshot 101 using optimistic concurrency — reader still queries snapshot 100 until explicitly moving to 101	• Iceberg uses optimistic concurrency control (OCC) with snapshot isolation or serializable isolation • writers validate parent snapshot at commit time • no locks, readers never block writers

Table 2: Hidden Partitioning and Partition Transforms

Transform	Example	Description
Identity	`PARTITION BY (region)` → partitions use exact values: `region=us-east`, `region=eu-west`	• Uses column value directly as partition value • equivalent to traditional Hive-style partitioning • useful for low-cardinality categorical columns

Table 3: Schema Evolution

Operation	Example	Description
Add Column	`ALTER TABLE orders ADD COLUMN promo_code STRING` — readers see `NULL` for old data; new data includes the column	• Adds column to schema • metadata-only operation (no data rewrite) • old files implicitly return `NULL` for the new column • fully compatible with existing data

Table 4: Time Travel and Snapshots

Feature	Example	Description
Time Travel by Timestamp	`SELECT * FROM orders FOR SYSTEM_TIME AS OF '2024-05-10 14:30:00'` — reads table state at that timestamp	• Queries table as it existed at a specific timestamp • Iceberg finds the nearest snapshot before or equal to the timestamp • useful for auditing, debugging, and regulatory compliance