New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
You're about to write a new ingest job and nobody told you which file format the data should land in.
What you'll have at the end
A script that saves one sample dataset as both CSV and Parquet, with the size and read-time difference written down.
You need
A representative sample of the data the new ingest job will write, and a script or notebook environment able to save and read files in more than one format.
Not covered
Picking a table-format layer such as Iceberg or Delta on top of the files, or choosing ORC over Parquet inside a pipeline that already lives entirely in Hive.
Decide if your dataset is big data
Check this first if the dataset barely reaches a few thousand rows: the size and speed gap measured here only starts to matter once a file gets genuinely large.
The small files problem, caught early
Once the new job writes Parquet, this covers the mistake that undoes the gain: writing one tiny file per event instead of a few larger ones.
Checked 25 Aug 2026
Part of the Big Data cookbook