Apache Kafka cookbook

12 recipes · 80 steps · about 65 minutes of reading

Publish and read your first Kafka message
You've never actually sent a message.
BeginnerRead 4 minDoing: about 20 minutes5 steps
Messages vanish when a Kafka producer isn't idempotent
Your producer call returns instantly, so you assume the message made it, until a broker blip proves you wrong.
Beginner to intermediateRead 5 minDoing: an afternoon6 steps
Commit Kafka consumer offsets so a crash doesn't skip or repeat messages
A crash mid-batch. Did it save, or reprocess?
IntermediateRead 5 minDoing: an hour or two7 steps
Pick a partition key that keeps one customer's events in order
Two updates for the same order arrive out of order downstream, and support wants to know why.
IntermediateRead 6 minDoing: an afternoon6 steps
Right-size a new Kafka topic's partitions
Someone on the team asks how many partitions the new topic needs, and you're about to just copy the number from another team's topic without checking it.
IntermediateRead 5 minDoing: a couple of hours6 steps
Why is my Kafka consumer group stuck rebalancing every few minutes?
Consumers keep dropping out and rejoining, throughput craters, and nobody touched the code.
IntermediateRead 5 minDoing: a couple of hours, plus time watching it hold steady afterward7 steps
One bad Kafka message stalls a partition with no dead-letter topic
Why did the whole partition stop moving over one bad row?
IntermediateRead 5 minDoing: an afternoon7 steps
Choose Kafka topic retention or compaction
Nobody can agree whether the new topic should keep seven days of history or just the latest value per key.
IntermediateRead 5 minDoing: an hour or two6 steps
Every consumer breaks without a Kafka Schema Registry check
A field type changed in last week's release, and three downstream consumers threw parsing errors at 2am.
Intermediate to advancedRead 6 minDoing: an afternoon7 steps
Decide whether a pipeline can tolerate duplicate Kafka messages or needs exactly-once
The finance pipeline is double-counting some events after every consumer restart. 'Just dedupe it downstream' worked for a while, but it isn't cutting it anymore.
Intermediate to advancedRead 6 minDoing: a day7 steps
Turn a nightly batch total into a live Kafka Streams window
The dashboard shows yesterday's numbers, and the business wants today's, updating as events arrive.
AdvancedRead 7 minDoing: an afternoon8 steps
Read a Kafka topic into a Spark Structured Streaming job
You already know PySpark, and hand-writing the same aggregation logic again as a raw Kafka consumer feels like starting over.
AdvancedRead 6 minDoing: a day8 steps

Cookbook badge

Finish all 12 recipes to earn this.

In these roadmaps

Data Engineerstep 11 of 16

Apache Kafka cookbook

Publish and read your first Kafka message

Messages vanish when a Kafka producer isn't idempotent

Commit Kafka consumer offsets so a crash doesn't skip or repeat messages

Pick a partition key that keeps one customer's events in order

Right-size a new Kafka topic's partitions

Why is my Kafka consumer group stuck rebalancing every few minutes?

One bad Kafka message stalls a partition with no dead-letter topic

Choose Kafka topic retention or compaction

Every consumer breaks without a Kafka Schema Registry check

Decide whether a pipeline can tolerate duplicate Kafka messages or needs exactly-once

Turn a nightly batch total into a live Kafka Streams window

Read a Kafka topic into a Spark Structured Streaming job

In these roadmaps

Apache Kafka cookbook

Publish and read your first Kafka message

Messages vanish when a Kafka producer isn't idempotent

Commit Kafka consumer offsets so a crash doesn't skip or repeat messages

Pick a partition key that keeps one customer's events in order

Right-size a new Kafka topic's partitions

Why is my Kafka consumer group stuck rebalancing every few minutes?

One bad Kafka message stalls a partition with no dead-letter topic

Choose Kafka topic retention or compaction

Every consumer breaks without a Kafka Schema Registry check

Decide whether a pipeline can tolerate duplicate Kafka messages or needs exactly-once

Turn a nightly batch total into a live Kafka Streams window

Read a Kafka topic into a Spark Structured Streaming job

In these roadmaps