New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
The finance pipeline is double-counting some events after every consumer restart. 'Just dedupe it downstream' worked for a while, but it isn't cutting it anymore.
What you'll have at the end
A written delivery-guarantee decision for the pipeline, with the transactional or idempotent-consumer config that actually delivers it
You need
A consumer that already reads Kafka events and writes each one into a downstream table, plus read access to that table's rows and to the consumer group's recent rebalance history.
Not covered
Choosing at-most-once delivery, which drops events outright: no pipeline touching money can accept losing data, so that option never enters this decision.
Messages vanish when a Kafka producer isn't idempotent
go there first if the numbers point at retry duplicates on the producer side rather than the consumer-redelivery pattern this recipe assumes.
Commit Kafka consumer offsets so a crash doesn't skip or repeat messages
for the deeper mechanics of exactly when a consumer should save its position, if an idempotent sink alone doesn't close the gap here.
Checked 25 Aug 2026
Part of the Apache Kafka cookbook