The Outbox Pattern: Boring, Ugly, Undefeated

Here is a bug I have written at least twice: save the order to the database, then publish an event to Kafka. Two operations, no shared transaction. Sometimes the save works and the publish fails, and now your order exists but no downstream system heard about it. Sometimes you publish first and the save fails, and now shipping is packing a box for an order that does not exist.

You cannot wrap a Postgres write and a Kafka publish in one transaction. Two different systems, two different commit protocols. What you can do is cheat, and the cheat is called the outbox pattern.

Instead of publishing to Kafka directly, you insert the event into an outbox table in the same database, inside the same transaction as the order itself. One commit, both rows, atomically. Either the order and its event both exist or neither does.

Then a separate relay process polls that table, publishes unpublished rows to Kafka, and stamps published_at. If the relay crashes after publishing but before stamping, it publishes again on restart. That gives you at least once delivery, which means your consumers need to handle duplicates, which they needed to do anyway because Kafka itself is at least once for most sane configs.

People resist this pattern because it feels crude. A table as a queue. Polling. A cron adjacent relay. Where is the elegance. There isn't any, and that is fine. Debezium can tail the WAL instead of polling if the extra moving part earns its keep, but plain polling every 200ms carried us to a few thousand events per second before we had to think harder.

The elegant alternatives are two phase commit, which nobody sane runs across Postgres and Kafka, or just accepting lost events, which you are probably doing right now without knowing it. Boring wins.