Feature Flags Are Distributed Systems Too and Yours Is Probably Broken

A feature flag looks like an if statement, which is the disguise it uses to avoid being reviewed like what it actually is: a distributed configuration system with its own consistency, availability, and lifecycle problems.

Start with the failure everyone hits eventually. Flag values get fetched from a service and cached locally. Twelve pods, twelve caches, refreshed on their own timers. Flip a flag and for the next 30 seconds your fleet disagrees about reality, some pods serving the new behavior, some the old. If the flag gates cosmetics, fine. If it gates which payment provider gets called, or which schema version gets written, that disagreement window is a data corruption machine. We learned to categorize flags by how much cross fleet skew they tolerate, and the intolerant ones get flipped via deploy, not via dashboard, because a rollout has ordering guarantees a cache refresh does not.

Next, what happens when the flag service is unreachable. Every SDK has a default for this, and if you have not chosen yours deliberately, it was chosen for you. Fail closed on a flag gating the new checkout means an outage at the flag service turns off checkout improvements, fine. Fail closed on a kill switch means the kill switch cannot kill anything during the exact chaos it exists for. We define the offline default per flag, in code review, as a required field.

Then the lifecycle problem, which is really a graveyard problem. Flags at 100 percent for a year are not flags, they are dead code with a network dependency. Ours now get owners and expiry dates, and CI nags when a flag outlives its date.

None of this argues against flags, they are how safe rollouts happen. It argues for respecting them as infrastructure, because they already are, whether respected or not.