What Happens When a Backend Dies Mid-Request?

Before building RelayForge, I'd have described "handling backend failures" as one feature. Health checks, retries, whatever you want to call it, it all blurs together into "the proxy notices a backend is dead and deals with it." Building both pieces separately, and then actually killing a backend mid-test to see what happened, made it clear they're not one feature wearing two names. They solve two different problems, on two different timescales, and neither one makes the other unnecessary.

Health checks: a periodic, independent opinion

A health check is simple: on a fixed interval, try to connect to each backend, and remember whether it worked.

func (s *Server) probeBackend(bs *backendState) {
    timeout := s.liveCfg().healthCheckTimeout
    conn, err := net.DialTimeout("tcp", bs.addr, timeout)
    isHealthy := err == nil
    if conn != nil {
        conn.Close()
    }
    bs.healthy.Swap(isHealthy)
}

That's genuinely the whole check, a plain TCP connect, nothing HTTP-aware about it, running in its own goroutine completely independent of whatever requests happen to be in flight at the time. A backend starts out assumed healthy, since nothing has proven otherwise yet, and only gets excluded from selection once a probe actually fails.

The thing worth being honest about: this is periodic, not reactive. If a backend dies one millisecond after the last successful probe, the proxy has no idea for up to a full interval. A request that happens to land on it in that window will still be sent there, and will still fail. Health checks are necessary, but they're not immediate, and pretending otherwise would be the kind of claim that only survives until someone actually measures it.

Retries: a one-time safety net, with real limits

Retry is a different mechanism entirely, operating at the level of one single request rather than the whole backend's standing in rotation. If forwarding a request to a backend fails, under specific conditions, try it again against a different backend, once:

func retryEligible(req *http.Request, a attempt) bool {
    if a.clientDisconnected {
        return false // nobody's waiting for the result anymore
    }
    if !a.dialed {
        return true // never reached any backend, nothing to duplicate
    }
    if req.ContentLength != 0 {
        return false // might have partially sent a body already
    }
    if !a.wroteFully && a.reused {
        return true // a pooled connection that was already dead, not a real failure
    }
    return isIdempotentMethod(req.Method)
}

That last condition is the one that actually matters most. A GET can be retried freely, doing it twice causes no additional harm. A POST, the classic example being something like POST /payment, cannot be retried blind. If the connection died after the backend had already processed the request, successfully, and the response just never made it back, retrying means asking the backend to do it again, and now you've potentially charged someone twice. RelayForge's answer is deliberately conservative: non-idempotent requests are retried only when nothing could possibly have reached the backend yet (the dial itself failed), never once a write might have landed. The ambiguous case, a response that goes missing after a backend actually did the work, is left to fail rather than silently risking a duplicate. That's not a limitation I plan to quietly fix later, it's the actual correct answer until there's a real mechanism (idempotency keys, something backend-side) to make a blind retry provably safe.

Killing a backend and actually timing it

Having both pieces built separately, I wanted to see what a real failure actually looked like end to end, not just reason about it. Two equal backends, round robin, a 200 millisecond health-check interval, one request every 200 milliseconds, and backend B killed with a real process termination partway through:

t=2849ms  req=9   code=200
t=3198ms  req=10  code=200
>>> killed backend-B at t=3198ms <<<
t=3665ms  req=11  code=200   (467ms gap, vs ~350ms baseline)
t=4015ms  req=12  code=200   (350ms gap, back to normal)

Every single request in that test returned 200. Not "mostly recovered", not "a brief error spike", zero non-200 responses across the entire run. Request 11, the first one to round-robin onto the now-dead backend after the kill, paid a small, almost entirely hidden cost: the connection attempt to the dead backend failed fast (a local refusal, not a slow timeout), and the retry against the surviving backend succeeded, adding about 115 milliseconds over the normal gap between requests. Every request after that avoided the dead backend entirely, which lines up exactly with the 200 millisecond health-check interval having already caught it by then.

Two mechanisms, two jobs

That timing is the whole point made concrete. Retry is what made failover invisible to the client, immediately, on the very next request, well before any health check had a chance to run. The health check is what stopped the proxy from paying the cost of that retry on every subsequent request, by taking the dead backend out of rotation entirely instead of discovering it was dead over and over. Take retry away, and that single request where backend B was still in rotation would have failed outright instead of quietly recovering. Take health checks away, and every single request that happened to round-robin onto the dead backend, forever, would pay that same small retry tax instead of just going straight to a backend that works.

Neither one is a replacement for the other. One is a fast, local, per-request reaction. The other is a slower, global, standing judgment about a backend's reachability. I'd genuinely assumed, before measuring it, that health checks alone would be "the" failure-detection mechanism and retries were more of a nice-to-have. The actual numbers said the opposite was closer to true: retry is what the client experiences during the gap, and the health check is what closes the gap back up.

Both mechanisms, along with the exact retry-eligibility rules and the test that produced these numbers, are on GitHub.