New: Cookbooks and AI ExplanationsStep-by-Step recipes to solve problems connected to Roadmaps and Cheat Sheets. Need more details? Use AI buttons for structured and simple explanations with concrete examples throughout the whole platform.Take a look
You already know PySpark, and hand-writing the same aggregation logic again as a raw Kafka consumer feels like starting over.
What you'll have at the end
A Structured Streaming job that reuses an existing PySpark aggregation, runs continuously against a live Kafka topic through an idempotent write, and is proven, offset for offset, to land on the same numbers a batch run over the same data would.
You need
A PySpark aggregation you already trust, built and tested against a static table, plus a Kafka topic carrying the same kind of records that table came from.
Not covered
Writing the aggregation logic itself, or sizing the topic's partitions; both are already settled coming into this.
Leans on
Turn a nightly batch total into a live Kafka Streams window
when the team wants a native Kafka Streams windowed count with no Spark cluster involved at all, not a PySpark job that reuses existing batch aggregation code.
Decide whether a pipeline can tolerate duplicate Kafka messages or needs exactly-once
when the open question is which delivery guarantee this pipeline actually needs, before deciding how the sink write should be structured.
Checked 26 Aug 2026
Part of the Apache Kafka cookbook