In 2020, a core network switch at Cloudflare partially failed, and a consensus-backed system degraded so badly that the dashboard ran 80 times slower than usual for six hours. No data was lost, but writes stayed blocked until the network was fixed. That incident is the starting point for Meerkat, the experimental consensus service Cloudflare uses to run its global control plane, and the reason it runs neither Raft nor Paxos but QuePaxa, an academic algorithm from 2023 with almost no production mileage.
The team laid out the reasoning in a post published on November 11, 2026. It is worth reading even if you will never write a consensus protocol: anyone running etcd, Consul or ZooKeeper lives with the same trade-offs.
The problem: the leader and the failure detector
Raft, Multi-Paxos and Viewstamped Replication share the same core idea: a single leader proposes, which stops multiple proposers from blocking each other (what the post calls destructive interference). While the leader is healthy and the network behaves, they decide in a single round trip.
The catch is that everything hinges on that leader. If it crashes, slows down or gets cut off, the system stalls until a new one is elected, and to get there the replicas have to guess that it failed. Over a wide-area network, a dead server and a late message look exactly the same. Cloudflare points to two consequences:
- False positives. A two-second latency spike makes the leader look dead, an election starts, and candidates fight for a majority and block each other. It is a metastable failure: the network recovers in two seconds, but the elections can drag on much longer.
- Slow even when right. The election timeout cannot go below network latency. With cross-continent round trips in the hundreds of milliseconds, detecting a dead leader takes around a second, and there are no writes in the meantime.
The etcd docs put numbers on it: the default election timeout is 1,000 ms, it should be at least 10 times the round-trip time between members, and the 50 s upper limit is meant for globally distributed clusters.
The fix: an optional leader nobody has to detect
QuePaxa (Tennage, Băsescu et al., 2023) changes three things:
- Proposers cooperate instead of competing. Every proposal carries a random priority; when a proposer sees a higher-priority one, it adopts it and spreads it as its own. Nobody can lock anyone else out indefinitely.
- The leader is optional. It always holds the top priority and is the only one that can decide in one round trip, but any replica can propose. If it goes down, the others carry on.
- There is no failure detector. Clients send the request to every replica, and non-leaders wait a short hedge timeout before proposing, to give the leader a head start. That timeout detects nothing, so it is not tied to latency: in Raft, a 5 ms timeout on a 100 ms network would freeze the cluster; in QuePaxa it just adds proposers.
Each round has four phases: propose and adopt (where the leader's fast path lives), spread, check and decide, and carry the best proposal into the next round. Safety comes from majority intersection: any two majorities share at least one replica, so if the leader reached a majority first, nothing else can be decided. In Cloudflare's Rust implementation, clients rather than replicas wait out the hedge timeout, which cuts the number of messages.
| Raft (etcd) | QuePaxa (Meerkat) | |
|---|---|---|
| Who proposes | Leader only | Any replica; the leader has priority |
| Leader goes down | Detect it, elect a new one; no writes meanwhile | Others keep deciding, more slowly |
| Key timeout | Election, ≥ 10 × RTT (1,000 ms default in etcd) | Local hedge, unrelated to RTT |
| Happy-path cost | 1 round trip | 1 round trip when the leader decides |
| Maturity | Decades in production | 2023 paper, under formal verification |
The results
In Meerkat tests with servers on nearly every continent, the leader makes over 95% of decisions in the first phase. When it does not, throughput briefly drops to at most a third, because three phases are needed instead of one. With the leader down, the decision rate falls to about 65% and goes back to 100% once it is replaced a minute later; Cloudflare says the system has never fully stalled. The team can also move leadership to another replica while the cluster is live.
The cost is spelled out too: the lower the hedge timeout, the more active proposers, the more bandwidth and the more failed rounds. Cloudflare chooses to pay in network traffic rather than lose writes.
What carries over
The takeaway is not "swap Raft for QuePaxa". It is that a failure-detection timeout is a bet on the network, and on wide-area links that bet is lost often. If you run etcd or Consul stretched across regions, the post explains why conservative election settings are not laziness: an aggressive timeout turns every latency spike into an election, and a failed election can feed the next one.
It is also a good model for taking research into production: Cloudflare starts from a real incident with numbers, picks a protocol for the properties it needs rather than its popularity, and offsets the risk with formal verification, simulation and fault injection.
What does not
Meerkat is experimental, internal and has no public code. QuePaxa lacks Raft's bench of battle-tested implementations, and Cloudflare is extending the original crash-stop design to support crash recovery, which is why it is formally verifying the result. If your cluster lives in one region with millisecond latencies, the problems QuePaxa solves barely touch you, and Raft remains the sensible choice.
Sources
- Choosing QuePaxa, an unusual consensus algorithm (Cloudflare, November 11, 2026)
- QuePaxa: Escaping the Tyranny of Timeouts in Consensus (SOSP 2023)
- etcd: Tuning