Big data storage formats are specialized file structures designed to efficiently store, compress, and query massive datasets in distributed computing environments. They fall into two primary paradigms: columnar formats (Parquet, ORC, Arrow) optimized for analytics with selective column reads and superior compression, and row-based formats (Avro, CSV, JSON) suited for write-heavy workloads and full-row access. Beyond basic file formats, open table formats (Delta Lake, Apache Iceberg, Apache Hudi, Apache Paimon, DuckLake) add a critical metadata layer that enables ACID transactions, schema evolution, time travel, and enterprise-grade reliability on top of immutable data files. As AI/ML workloads grow, a new generation of formats (Lance, Nimble, Vortex) targets vector search, random access, and wide-table feature engineering — use cases where Parquet shows its age.
What This Cheat Sheet Covers
This topic spans 25 focused tables and 193 indexed concepts. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
A jump-to index of every table row in this cheat sheet.
An interactive map of every table and concept in this topic.
Table 1: Storage Paradigms
The fundamental choice between columnar, row-based, and hybrid storage shapes every other decision in a data platform. Most modern lakehouses combine columnar files on object storage with a table format metadata layer.
| Paradigm | Example | Description |
|---|---|---|
SELECT revenue FROM salesreads only revenue column | • Stores data by column rather than row • enables selective column reads, superior compression (10-100x vs row formats), and vectorized processing • Ideal for OLAP/analytics workloads | |
INSERT INTO users VALUES (...)writes entire row at once | • Stores complete records sequentially as rows • optimized for transactional writes, full-row retrieval, and frequent updates • Better for OLTP; poor compression vs columnar |