Real-time streaming is a bill most teams never needed

Every data platform pitch deck contains the word "real-time" and every finance team contains a person who checks the revenue dashboard on Monday mornings. Between those two facts lives a great deal of wasted money.

Streaming is a genuine requirement for a short list: fraud scoring, operational alerting, anything where a machine acts on the event within seconds. It is a genuine waste for the much longer list where the consumer is a human with a meeting cadence. The cost difference is not marginal. A streaming pipeline is an always-on distributed system with state, ordering, backpressure and an on-call rotation. A batch job is a query with a schedule. One of these fails at 3 a.m. in interesting ways; the other reruns.

the arithmetic vendors skip

Freshness has a price curve, and it is not linear. Moving a pipeline from daily to hourly is usually cheap. Hourly to minutes means new infrastructure. Minutes to seconds means a different team. The honest question is never "would fresher be better", because the answer is always yes and always worthless. The question is which decision changes at each freshness tier, and whether that decision's value covers the tier's cost.

Boring evidence that batch carries serious workloads: a public insurance data warehouse case study runs 150+ daily extracts across five external companies on scheduled batch, feeding regulatory and financial reporting. Nobody's fraud model lives there, and nothing about it would improve if it streamed.

how streaming sneaks in anyway

The route into the estate is rarely a decision; it is a default. An ingestion vendor turns on "real-time sync" because the toggle demos well. A platform team adopts a message bus for one legitimate use case, and every subsequent pipeline gets built on it because the bus is there and the team knows it. Two years later the architecture review finds nightly finance numbers riding an exactly-once streaming topology, and nobody can name the decision that required it, because there was no decision, only drift with momentum. The audit question that surfaces this is simple: for each streaming pipeline, name the consumer that acts within a minute. Machines qualify. Meetings do not.

the middle tier nobody markets

The unfashionable answer to most freshness requests is micro-batch: the same batch job, scheduled every five or fifteen minutes, with idempotent writes and watermark bookkeeping. It preserves the property that makes batch operationally kind, reruns are the recovery story, while covering the "the morning meeting wants numbers from an hour ago" tier that gets misfiled as a streaming requirement. Modern warehouses and table formats handle fifteen-minute loads without ceremony. The reason no one markets this is that there is nothing to sell; it is the infrastructure you already own, run more often. When a stakeholder says real-time, the productive translation is usually "before my meeting", and a schedule can attend meetings.

A budgeting rule that keeps estates honest: every pipeline carries its freshness tier as metadata, daily, hourly, micro-batch or streaming, and every promotion up a tier names the decision that funds it and the person who owns that decision. The register takes an hour to create and turns the next "can we make this real-time" conversation from an architecture debate into a line-item request, which is the correct genre for it.

verdict

Default to batch, promote individual pipelines to streaming when a named decision demands it, and treat any architecture that begins with a message bus and works backward to the use case as marketing. Fresh data nobody acts on is the most expensive kind of stale.