The benchmark you didn't run is the one that matters

Every platform we have reviewed this year arrived holding a benchmark. The numbers were splendid. They were also, without exception, produced on hardware we do not own, tuned by engineers we cannot hire, against data shaped nothing like ours.

This is not fraud. It is selection. A vendor benchmark is the vendor's best day at the office, preserved in amber and printed on the pricing page. Nothing in the genre requires the workload to resemble yours, and the genre politely declines the requirement. Goodhart's law covers the rest: once a query suite becomes a sales target, it stops measuring the platform and starts measuring the effort spent on the suite.

We keep re-learning this at our own desk. When we compared orchestrators, the published throughput figures reordered themselves the moment our own DAGs arrived, which have the poor manners to contain joins. When we weighed DuckDB's single-node case, the interesting number was never the TPC-H run; it was how far a laptop got on our actual files.

So, the journal's standing advice, at memo length. Take your five worst queries, your ugliest table and one week of real arrival patterns. Run them on the trial tier. Measure the bill while you measure the speed, because a benchmark that omits cost has told half a story. Then set the result beside the number on the slide and enjoy the silence in the room.

The chart above is illustrative, as our charts say when the shape is the argument. The shape holds. The published bar is tall, yours is shorter, and the gap is not the vendor lying. It is the distance between their best day and your ordinary one. You are buying the ordinary one.

Verdict: a benchmark you did not run is an advertisement. Run yours.