Change data capture, the primer we keep re-explaining

Every few months someone proposes hammering the production database with SELECT * WHERE updated_at > ? every five minutes, and every few months the same conversation follows. This primer is that conversation, written down.

Change data capture reads the database's own transaction log (the WAL in Postgres, the binlog in MySQL) and emits every insert, update and delete as an event. The canonical open-source implementation is Debezium, and the approach's advantages over polling are structural, not incremental. The log sees deletes, which polling never does. It sees every intermediate state, not whichever row version your five-minute window happened to sample. And it costs the source almost nothing, because the database was writing that log anyway.

where CDC earns its complexity

Replicating operational data into the warehouse without hurting the operational database. Feeding search indexes and caches that must not serve deleted rows. Event-driven integrations where "the order changed" is the product. If your requirement is a nightly snapshot for a weekly report, a batch extract remains the honest tool (the batch defense applies).

the bill nobody quotes

CDC is a distributed system with the database's most private format at one end. Schema changes upstream become your problem downstream, in real time. Snapshots for new tables, exactly-once delivery into the warehouse, connector upgrades that track database versions: each is manageable, none is free, and together they are why "we'll just read the binlog" is a sentence that precedes a bad quarter.

buy or run, the only open question

The architecture debate is settled enough that the remaining decision is operational: run the open-source connectors yourself or pay a managed service to absorb the pain. The pricing of the managed tier is per-row moved, which reads cheap in the demo and compounds badly on high-churn tables; the self-run tier is priced in engineer attention, which reads free and never is. The tiebreaker we suggest is the upgrade log. Read three months of connector release notes for your database version and ask who on the team would have shepherded each of those upgrades through staging. If a name comes to mind and that person has slack, self-run is honest. If the name that comes to mind is already on-call for four other systems, the per-row fee is not a margin grab, it is staffing arbitrage, and paying it is cheaper than learning the WAL format during an incident.

verdict

Adopt for replication and event-driven cases; skip for report freshness theater. And budget the operational line before the first connector ships, because the log never stops and neither does the pipeline that reads it.