New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
A crash mid-batch. Did it save, or reprocess?
What you'll have at the end
A consumer that commits offsets after each processed batch, tested by killing it mid-batch and checking exactly what replays
You need
A Kafka consumer that already connects to a topic and pulls records, plus a place downstream, such as a table or key-value store, where a record can be written by its own id.
Not covered
Deciding whether your pipeline can tolerate occasional duplicates at all, or building true exactly-once delivery with Kafka's transactional APIs.
Decide whether a pipeline can tolerate duplicate Kafka messages or needs exactly-once
go there first if even a harmless-looking duplicate write isn't acceptable, to weigh real exactly-once processing against tolerating replays.
Messages vanish when a Kafka producer isn't idempotent
this recipe assumes messages already arrive reliably; that one covers the producer side, a different way messages go missing.
Checked 25 Aug 2026
Part of the Apache Kafka cookbook