Two Leaders Walk Into a Cluster: My First Real Split-Brain

September 09, 2026 / systems / 371 words

Everyone learns Raft from the paper or the pretty visualization site. I learned it at 11pm on a Tuesday when our config service started returning two different values for the same key depending on which node you asked.

The symptom was subtle. No pages, no crashes. Just a slow trickle of "hey, my feature flag flipped back?" messages in Slack. Feature flags don't flip themselves, so I SSH'd into the bastion and asked etcd what it thought was going on.

An etcdctl endpoint status table showing two nodes both reporting leader true on different raft terms, 842 and 839, one endpoint timing out, and pings of around 400 milliseconds inside the datacenter.

Look at that table. Node A thinks it's the leader on term 842. Node C also thinks it's the leader, on term 839. Two leaders, two terms, and a fourth endpoint timing out entirely. In theory Raft makes this impossible, a node on an older term should step down the moment it hears from anyone on a newer one. The catch is "the moment it hears from anyone." Node C wasn't hearing from anyone.

The ping told the real story. Zero packet loss, but 400ms round trips inside the same datacenter. A network engineer later confirmed a misbehaving top-of-rack switch was queueing packets long enough that node C kept missing heartbeats, calling elections nobody answered, and serving stale reads to the handful of clients unlucky enough to be routed to it. Not a true partition. Worse, a gray failure, where the network is technically up and practically useless.

Two things I took away from this.

First, stale reads are a client problem as much as a server problem. Our clients were using serializable reads for speed, which means "read from whatever node you're connected to, no quorum check." That's a fine tradeoff until it isn't. We moved the flag-critical paths to linearizable reads and ate the latency.

Second, "is it up" is the wrong health check. Every node responded to ping. Every node passed its HTTP health endpoint. What we actually needed to alert on was raft term divergence and leader count across the cluster, signals from inside the consensus protocol, not around it.

Once the switch got yanked, node C rejoined as a follower, terms converged at 843, and the cluster acted like nothing happened.

etcdctl member list and cluster health output after the bad switch was removed: all three nodes started and healthy, committing proposals in 9 to 15 milliseconds.

Raft did its job. The paper just doesn't have a section for switches that lie.