Batch or streaming: pick the shape of a feed

Someone in planning says we need this in real time, and you're the one who has to say what that actually costs.

Intermediate to advancedRead 7 minDoing: about an hour7 steps

▶ Start

What you'll have at the end

A one-page decision, batch or streaming, for one real data feed, naming the actual latency requirement it has to meet.

Before you start

You need

A specific feed someone wants sped up, plus access to its own history: how fast the thing it tracks actually moves, and how often whatever acts on it gets to act.

Not covered

Building the pipeline that actually moves the data once the shape is picked: this recipe only settles what shape it should be.

Leans on

Big Data Cheat SheetNames the real engines this decision would point you toward once the shape is picked.
Stream Processing Fundamentals Cheat SheetCovers the plumbing an always-on feed would actually add, the exact cost this recipe weighs against the benefit.

7 steps

Where to go next

Add a freshness and volume check so bad data never reaches a dashboard

once this feed is live, that recipe adds the automatic check confirming it actually keeps meeting the interval you just wrote down.

Checked 25 Aug 2026

Part of the Big Data cookbook