When a stretched quorum gets the jitters: a WAN-raft post-mortem

For about six hours one afternoon, two of our sites — the studio’s own site and one of our apps — served 503 Service Unavailable. No deploy had gone out. Nothing had been touched. The machines were up, the orchestrator was up, and yet the apps were simply gone.

This is the story of why, and it starts somewhere you might not expect: with a decision to make the cluster more reliable.

The expansion that changed the physics

We run our workloads on Nomad, with Consul underneath for service discovery. It began life on a single server. A single server is a single point of failure, so — sensibly — we grew the control plane into a proper three-voter quorum: the original box plus two more, each at a different hosting provider. Three voters means one can die and the cluster keeps making decisions. Textbook high availability.

Here’s the subtlety nobody tells you at the whiteboard: raft, the consensus protocol under both Nomad and Consul, was designed for a datacenter LAN. Its default timings assume the servers talk to each other in well under 10 milliseconds, over a network that essentially never drops a packet. That assumption is baked into a single knob, raft_multiplier, which defaults to 1 — meaning one-second heartbeat and election timeouts.

The moment we spread those three voters across three providers, raft stopped running on a LAN. It started running over the public internet — tens of milliseconds of latency, variable jitter, and the occasional lost packet. We kept the LAN defaults. We had, without realizing it, wired a sprinter’s reflexes into a system that now lived on a bumpy road.

Why it got jittery

Raft elects one leader; the followers expect a heartbeat from it on a tight schedule. Miss a few heartbeats and a follower assumes the leader is dead and calls an election. With one-second timeouts, it doesn’t take much of a network hiccup to miss enough heartbeats.

One of our three providers had a genuinely unstable uplink. A per-hop trace told the story in one line — the very first hop out of that node:

1.|-- <provider edge>   Best 1.3ms   Avg ~25ms   Worst 166ms   StDev ~20ms

Best case 1.3 ms, worst case 166 ms, on the first hop. That’s not packet loss — that’s a firehose of jitter injected right at the node’s edge, and everything downstream inherits it. Layer intermittent bursts of real packet loss on top (we measured 10–13% during bad spells), and the picture is complete: a raft leader whose heartbeats sometimes arrived late, sometimes not at all.

The result was chronic churn. On its worst day, that one node called 229 leader elections in 24 hours. The quorum never actually fell apart — raft’s pre-vote mechanism kept rejecting the doomed candidacies — but it was permanently twitchy. A user-visible failure wasn’t an if, it was a when.

Interestingly, Consul on those same nodes rode it out far better. That’s because Consul’s raft_multiplier defaults to 5, not 1 — five-second timeouts, five times more tolerant. Same network, same jitter, dramatically calmer. That contrast was the tell: our Nomad control plane was the fragile one, purely because of one default value.

The disaster nobody could see brewing

The churn had been running for weeks. Nothing paged us, because nothing was down — the quorum stayed up. But the instability was quietly accumulating debt:

We later found a service whose Nomad job had silently been garbage-collected weeks earlier. Its container happened to keep running — so there was no outage — but Nomad had stopped managing it entirely: nothing was watching it, and nothing would have restarted it if it had fallen over. We simply had no idea it had become unmanaged.
A near-identical mass-disruption event had already happened days earlier. That time, everything happened to recover cleanly, so it passed unnoticed.

And there was a trap waiting that made the blindness worse. The load balancer in front of the apps answered its health check — /healthz returned a cheerful 200 OK — even while the app behind it was completely dead. That 200 came from the load balancer’s own built-in monitor endpoint, not from the application. Every naive “is it up?” check said yes. The building was on fire and the smoke detector was wired to a green light.

The afternoon it finally broke

Eventually the dice came up wrong. A network burst hit at the worst moment. It wasn’t just the raft leader that stumbled this time — a client node’s heartbeat to the servers blew past its deadline. The default grace period for a client to check in is ten seconds; the round-trip latency spiked to 14.9 seconds.

To Nomad, a client that hasn’t checked in is a client that has died. And when a client “dies,” the scheduler does the correct, brutal thing: it declares every workload on that node lost and re-places all of it elsewhere. Two client nodes crossed the line at once. In an instant, essentially everything was marked lost and thrown back into the scheduler to be rebuilt from scratch — all at the same time, colliding over ports as old containers lingered.

That’s the mass re-placement. But re-placement, in theory, is recoverable — the scheduler should just start everything again. So why did two of our sites stay dead for six hours?

Why some apps refused to come back

Here’s the part that turned a blip into an outage.

Some of our jobs fetch their binary from a registry in a prestart step — a little curl that runs before the app starts. Normally invisible. But these prestart fetches fired during the network burst, when packets were dropping. The curl failed (exit 56 — connection reset). Nomad retried it a few times, still inside the same bad window, and exhausted its restart attempts. The deployment was configured to auto-revert on failure — but the revert had nothing healthy to revert to, so it failed as well. The job landed in a dead state with zero pending evaluations.

That last detail is the killer. A dead job with no evaluations is not something the scheduler will ever retry on its own. From Nomad’s point of view, the job is done — deliberately stopped, nothing left to do. It will sit there dead forever, silently, until a human resubmits it.

And the jobs that didn’t have a fetch-on-prestart step? They just restarted and came back fine. The differentiator wasn’t importance or luck — it was whether a job had to pull something over the network at the exact moment the network was on fire.

The immediate recovery was simply resubmitting the dead jobs by hand. They came up first try — the binaries, tokens, and config were all still valid. The jobs hadn’t broken; they’d merely ceased to exist as scheduling objects, and needed to be told to exist again.

The fixes

Three changes, each aimed at a specific link in the chain:

raft_multiplier = 5 on all three servers. This is the big one — it scales raft’s timeouts from one second to five, so ordinary WAN jitter no longer looks like a dead leader. It’s exactly the value Consul already used to ride out the same network.
heartbeat_grace = 60s. This widens how long a client can go quiet before it’s declared dead. A 15-second latency spike no longer nukes every workload on a node.
Retry-with-backoff on the prestart fetches (curl --retry 5 --retry-all-errors --retry-max-time 120). Now the binary fetch rides out a network burst instead of dying on the first dropped packet — closing the “fetch failed → job stuck dead” gap.

We’re honest about what this does and doesn’t fix. It makes the cluster tolerant of the bad uplink; it does not repair the uplink itself. That jittery first hop is a physical-layer problem at one provider, and the only real cure there is a provider ticket or replacing that voter. The tuning buys us resilience; it doesn’t change physics.

Chasing the physics — the provider ticket

Tolerance isn’t a cure. The packet loss was still real, one node was still the weak leg, and a bad enough day could still bite — so we went after the physical defect directly, which meant a ticket to the provider. And that’s its own small skill, because “we’re seeing packet loss” earns a “no fault found” the instant a technician tests during a calm minute.

The evidence that actually gets a provider to act on it is the thing you can reproduce on demand. For us that wasn’t the intermittent loss — it was the latency jitter on the provider’s own first hop: best case ~5 ms, worst case ~160 ms, on a clean hundred-packet sample, every single run. That variance is born on their edge and inherited by every hop downstream. We backed it with timestamped loss bursts from the outage itself, plus a control: the link between our other two providers was flawless — 0% loss, sub-10 ms, no jitter — over the same window, which isolates the fault to one segment and forecloses “it’s your config.”

The measurement itself, because a number is only as good as how you got it: the tool is mtr — traceroute and ping fused into one — run from the node, a hundred ICMP probes a shot (mtr -r -n -c 100 <target>). It keeps per-hop min/max/std-dev latency, and the whole story is in hop 1 — the provider’s own edge router, the first thing past our machine. A cleaned-up version of that first row:

 1.|-- <provider edge>   Loss 2.0%   Best 4.2 ms   Worst 161.3 ms   StDev 29.6 ms

A healthy first hop, being the provider’s own edge router — effectively next door — should be sub-millisecond and, above all, stable: best, average, and worst within a hair of each other. This one runs a 4 ms best to a 161 ms worst — a ~30 ms standard deviation — which means the jitter is born on their edge, not somewhere out in the wider internet you could argue about, and every hop below it just inherits the same wobble. Two method notes make it stick: we ran it from both of our machines on that provider and got the same signature every time (that’s the “reproducible on demand”), and we measured latency with ICMP but loss with a separate ping sweep — mtr’s TCP/UDP modes report a bogus 100% loss even on a perfectly healthy link, so they’re useless for this.

That framing landed: the provider took the ticket on and is working to fix the uplink. Two things to take from it. Lead with what you can demonstrate live, not what you happened to catch once — a jitter figure you can reproduce beats a loss spike you saw at 3 a.m. And keep your baseline: “the provider says it’s fixed” and “our numbers say the jitter is gone” are different claims, and we kept the traceroutes precisely so we can tell which is true when they close the ticket.

Why we now watch from the outside — the blackbox exporter

The most damning fact of the whole incident wasn’t the outage itself. It was that two public sites were dead for six hours and nothing told us — we found out by looking, not by being paged. So the structural fix — more important than any timeout value — was monitoring that would actually catch this.

The critical realization: you cannot monitor a dead system from the inside. Consider what was available:

Application metrics (the “white-box” approach — the app exports its own health, request counts, latencies). Useless here: a job that has ceased to exist exports nothing. There’s no process to ask. Silence from a metrics endpoint is ambiguous — is it healthy and quiet, or gone?
Internal health checks. We saw how that ended: the load balancer’s own endpoint returned 200 while the app behind it was dead. An internal check can be fooled by any layer that answers on the app’s behalf.

The answer is black-box monitoring: probe the real, public URL from the outside, exactly like a visitor would, and judge it only by what actually comes back. That’s what Prometheus’s blackbox exporter does. We picked it for reasons that map directly onto the failure we’d just lived through:

It tests the thing users test — the public URL over real HTTPS, end to end, through every layer. If a visitor would get a 503, the probe gets a 503. No internal component can lie on the app’s behalf.
A dead job can’t hide. There’s nothing to self-report; the probe simply gets no valid response and fails. Absence is detected, not assumed-healthy.
It follows redirects, so an auth-gated dashboard that bounces through a login (302 → 200) reads as up, while a genuinely broken one reads as down — no false alarms, no false all-clears.
It’s boringly standard. Blackbox exporter + Prometheus for the probes and alert rules, Alertmanager for delivery to email. All parts we already understood, nothing exotic to babysit.

Now every public site is probed continuously from outside. If any of them returns something other than success for three minutes, an alert lands in the inbox — with the SSL-certificate-expiry checks thrown in for free while we were at it. That blind spot is closed: the first time a site goes dark now, we know in minutes, not never.

The lessons, distilled

Stretching a quorum across providers inherits the WAN’s physics. High availability on paper can be lower availability in practice if you keep datacenter defaults on an internet-distance cluster. Tune the timeouts for the network you actually have.
A resilient system still needs its dependencies to be resilient at the worst moment. Our jobs were fine; a fetch that ran during a burst is what stranded them. Assume every network call can happen during a storm, and make it retry.
You can’t watch a dead thing from inside it. Health signals that live in the same process, box, or proxy as the app can vanish or lie precisely when you need them. Probe from the outside, judge by what a user would see, and alert on that.

None of the individual pieces were exotic. A default left unchanged, a network worse than assumed, a retry that wasn’t there, and a health check that pointed at the wrong thing — each small, and together six hours of downtime. The cluster is calmer now, and more importantly, the next time something does break, we’ll be the first to know instead of the last.