Stream processing is the continuous, real-time computation over unbounded data flows, enabling organizations to analyze and act on events as they occur rather than waiting for batch windows. It sits at the intersection of data engineering and real-time systems, powering everything from fraud detection to live dashboards. Understanding stream processing requires mastering the trade-offs between latency and completeness, the semantics of time in distributed systems, and the guarantees your application can make about correctness. The key insight: streaming is batch where the batch never ends—windowing, watermarks, and stateful aggregation let you impose structure on infinite flows while handling the messiness of real-world event arrival patterns.
What This Cheat Sheet Covers
This topic spans 14 focused tables and 88 indexed concepts. Below is a complete table-by-table outline of this topic, spanning foundational concepts through advanced details.
Table 1: Core Processing Models
| Model | Example | Description |
|---|---|---|
spark.read.parquet("hdfs://data/") .groupBy("id").count() | • Processes bounded datasets with defined start and finish • optimized for high throughput and historical accuracy over low latency. | |
stream.keyBy("userId") .window(TumblingTime.of(minutes(5))) .sum("amount") | • Processes unbounded data continuously as it arrives • optimized for low latency and real-time action over resource efficiency. | |
spark.readStream.trigger( Trigger.ProcessingTime("10 seconds")) | • Collects events into small time-bound batches (e.g., 10s intervals) • balances latency and throughput by processing mini-batches instead of individual records. |