Backpressure or Bust: What Happens When You Ignore Queue Depth

A queue in front of a slow consumer feels like a solution. It is actually a decision you have deferred, and the queue is where the decision compounds interest.

Here is what our lag looked like the day one consumer group fell behind:

Partition 1 sitting 320 thousand messages behind while its siblings hummed along. One hot partition key, one drowning consumer, and because we had no backpressure anywhere, producers kept cheerfully stuffing the topic at full rate. Nothing was down, so nothing alerted. Events were just getting older. By the time someone noticed, the freshest message on that partition was 40 minutes stale, which for the feature involved meant users were seeing decisions made on data from a different reality.

Backpressure is the missing feedback loop, some way for the slow end of a pipeline to tell the fast end to ease off. Without it, pressure has to go somewhere, and it goes into the queue as unbounded lag, or into memory as an OOM, or into your data as silent staleness. With it, the pressure surfaces at the edge, where you can make an actual decision: slow the producers, shed low value load, or spin up capacity.

What we changed, in rough order of payoff. Alert on lag and on message age, not just service health, a healthy consumer working through ancient messages is not healthy. Bounded internal buffers everywhere, block or drop when full, chosen explicitly per queue instead of defaulting to unbounded. Fixed the hot key by salting the partition key for the one aggregate causing it. And load shedding at ingest for the traffic tier we could afford to drop.

Every unbounded queue in your system is a promise that consumers outrun producers forever. Forever is a long time.