A single Cloudflare DNS zone can hold more than 20 million records, and until recently, flipping Universal SSL on or off was enough to make the Zone Builder rebuild the whole thing. After fixing that, Cloudflare's DNS team cut the service's average memory from 49 to 7 GiB and halved the time it takes for a change to reach its configuration store, as Alex Fattouche explains on the Cloudflare blog on November 2, 2026.
The problem: more events, ever-bigger zones
The Zone Builder turns database state into the keys authoritative DNS servers read and writes them to Quicksilver, Cloudflare's distributed key-value store. In 2022 it handled about 250 record changes per second; today it is around 700, plus another 100 events such as zone setting changes.
Volume wasn't the real issue. The kind of work was. Back in 2022 the team had already made a record change rebuild only that record (150 times faster than a full build on average). But zone setting changes still triggered a full build: read every record, encode it, diff it against Quicksilver, and hold the entire zone in memory. For the largest zones, that doesn't even fit.
The fix: a planner that only cuts work when it can prove it is safe
At the core is a planner that picks one of five build types: specific names only, names plus shared metadata, apex plus metadata, a zone purge, or a full zone. One simple rule governs it: every key a targeted build writes must be byte-for-byte identical to what a full build would write. If that can't be proven, it runs the full build. An ambiguous, incomplete or unknown event never turns into less work.
Getting there meant solving several problems at once:
- Events that say what changed. The producer now includes the modified fields and settings, with old and new values when available. Some database functions only bump a timestamp; those events still trigger a full build until the caller behind them is identified.
- Planning twice. The event yields a conservative plan, and right before running, the builder reads a fresh snapshot from read replicas. The plan can grow into a full build, become a purge or be skipped. Because Kafka delivery and replication move independently, the aggregator delays events and the database rejects the read if the replica is behind.
- One scheduler. Record work and full-zone work used to run on separate schedulers that couldn't combine work. Now a single scheduler merges everything queued for a zone, and merging can only add work: if any event needs the whole zone, the combined build is a full one, and a purge overrides every rebuild.
- Never losing a message. The scheduler tracks DNS work and unacknowledged Kafka messages separately. With soft committing, builds can finish out of order, but the offset only advances once every earlier message is done. After a restart they would rather redo work than drop a change.
There is one deliberate exception: above a record-count threshold, the Zone Builder never escalates to a full build, because it would never finish and would stall its whole Kafka partition while retrying. Instead it rebuilds the apex, the metadata and the affected names, and the rare leftover cases are fixed by hand.
How they proved it before shipping
This is the most reusable part of the story. Beyond tests that compare a targeted build with a full one from the same state, byte for byte, the rollout ran a full build alongside every targeted build. And every release goes through a mirror Zone Builder that consumes the same production Kafka stream and reads the same database, but writes to a separate Quicksilver that serves no traffic. A comparer checks 20% of zones; the screenshot they share shows a 0.31% diff rate across 234,313 zones, almost all of it timing skew between the two environments.
The rollout was keyed on a hash of the zone name, not the zone ID. Every zone sharing a name (and therefore the same metadata key) landed on the same scheduler, so two schedulers never wrote the same key. Free zones went first, paid zones after.
Results
| Metric | Before | After |
|---|---|---|
| Average Zone Builder memory | 49 GiB | 7 GiB |
| Peak memory | 79 GiB | 12 GiB |
| Worst p99 delay spike (24 h) | 14.6 min | 1.21 min |
| Median delay to Quicksilver | ~7.7 s | ~3.3 s |
| Kafka consumption | bursts of up to 1,550 msg/s | tracks production (600–800 msg/s) |
Over 24 hours, 92.95% of builds avoided rebuilding the full zone; full builds dropped by about 40%, and the ones left average 0.35 s because they mostly hit new, small zones. The median halved because cheaper builds let the team shorten two safety waits: the minimum interval between rebuilds of the same zone, and how long the aggregator holds each event so replicas can catch up.
What I take away
The first lesson is the equivalence contract. Optimizing anything that rebuilds state (a controller, a config generator, a cache) is risky because the typical failure isn't an error, it's stale data nobody notices. Requiring the incremental path to produce exactly what the full path would, and falling back to the full path whenever in doubt, turns the optimization into something you can verify.
The second is a mirror environment fed by real traffic. Tests can't cover every customer combination; a parallel consumer of the same stream writing to an isolated target, with a sampling comparer, can.
The third is about measurement. Their histogram had buckets at 1, 5, 10 and 30 seconds, and p99 came out near 30 s; with a native, fine-grained histogram the real p99 is around 12 s and p90 about 5.5 s. If your latency SLOs rely on percentiles computed over coarse buckets, you are probably reading inflated numbers.
What doesn't carry over
Quicksilver and the Zone Builder are internal, and the scale (zones with tens of millions of records, hundreds of events per second) is far beyond most teams. What you can copy are the patterns: plan from what the event says, verify against fresh state before writing, and never commit an offset until everything before it is done.
Sources
- How we halved DNS build delay and cut memory use by 85%, Cloudflare Blog, November 2, 2026.
- How we improved DNS record build speed by more than 4,000x, Cloudflare Blog, May 25, 2022.