Big Data cookbook

8 recipes · 50 steps · about 47 minutes of reading

Choose Parquet over plain CSV
You're about to write a new ingest job and nobody told you which file format the data should land in.
BeginnerRead 4 minDoing: about an hour6 steps
Decide if your dataset is big data
Someone says you need Spark for this, and you're not sure the file is even big enough to need it.
BeginnerRead 5 minDoing: 15 to 20 minutes the first time, a few minutes once it's routine6 steps
SQL or NoSQL: pick a database for your next feature
Product wants this feature live today. SQL table, or a document store?
Beginner to intermediateRead 6 minDoing: twenty minutes6 steps
The small files problem, caught early
Your new job writes one file per event, and six months later the data lake is crawling.
Beginner to intermediateRead 6 minDoing: a couple of hours6 steps
Query a folder of Parquet files directly instead of loading them into a warehouse
You need one answer from last month's exported Parquet files, and standing up a warehouse table for it feels like overkill.
IntermediateRead 5 minDoing: under an hour6 steps
Catch a skewed join key before it stalls a distributed job
A dashboard that always finishes in ten minutes. Stuck at ninety percent for the last twenty, and nobody touched the code.
IntermediateRead 5 minDoing: under an hour6 steps
Batch or streaming: pick the shape of a feed
Someone in planning says we need this in real time, and you're the one who has to say what that actually costs.
Intermediate to advancedRead 7 minDoing: about an hour7 steps
Add a freshness and volume check so bad data never reaches a dashboard
A pipeline finished green three days ago, and nobody noticed the source stopped sending new rows until the numbers looked flat.
Intermediate to advancedRead 9 minDoing: an afternoon7 steps

Cookbook badge

Finish all 8 recipes to earn this.

In these roadmaps

Data Engineerstep 7 of 16

Big Data cookbook

Choose Parquet over plain CSV

Decide if your dataset is big data

SQL or NoSQL: pick a database for your next feature

The small files problem, caught early

Query a folder of Parquet files directly instead of loading them into a warehouse

Catch a skewed join key before it stalls a distributed job

Batch or streaming: pick the shape of a feed

Add a freshness and volume check so bad data never reaches a dashboard

In these roadmaps

Big Data cookbook

Choose Parquet over plain CSV

Decide if your dataset is big data

SQL or NoSQL: pick a database for your next feature

The small files problem, caught early

Query a folder of Parquet files directly instead of loading them into a warehouse

Catch a skewed join key before it stalls a distributed job

Batch or streaming: pick the shape of a feed

Add a freshness and volume check so bad data never reaches a dashboard

In these roadmaps