Round Robin vs Least-Connections, Benchmarked Instead of Assumed

Every load balancer you've ever configured offers you a choice of algorithm, and round robin is almost always the default. It's simple, it's fair in an obvious way, and when every backend is roughly the same speed, it's basically fine. I'd configured that default dozens of times without ever being in a position to say, concretely, when it stops being fine. Building the load-balancing layer of RelayForge myself, and then benchmarking it against the alternative, finally gave me a real answer.

Round robin: simple, and blind on purpose

Round robin's whole design is that it doesn't look at anything. It keeps a counter, and each request gets the next backend in the list, wrapping back to the start when it runs out:

func (s *Server) pickRoundRobin(list []*backendState) (*backendState, bool) {
    n := len(list)
    for i := 0; i < n; i++ {
        idx := int((atomic.AddUint64(&s.next, 1) - 1) % uint64(n))
        if list[idx].healthy.Load() {
            return list[idx], true
        }
    }
    return nil, false
}

That's the entire algorithm. An atomic counter, a modulo, a healthy check so a dead backend gets skipped. It doesn't ask how busy a backend currently is, how fast it's been responding, or anything else about its actual state. Every backend gets exactly the same share of traffic, by construction, which is precisely the property that makes it both simple and, under the wrong conditions, a bad idea.

Least-connections: pick whoever's least busy right now

The alternative looks at one thing round robin ignores entirely: how many requests each backend is currently handling, and sends the next request to whichever has the fewest:

func (s *Server) pickLeastConnections(list []*backendState) (*backendState, bool) {
    n := len(list)
    start := int((atomic.AddUint64(&s.next, 1) - 1) % uint64(n))
    var best *backendState
    var bestCount int64
    for i := 0; i < n; i++ {
        bs := list[(start+i)%n]
        if !bs.healthy.Load() {
            continue
        }
        count := atomic.LoadInt64(&bs.active)
        if best == nil || count < bestCount {
            best, bestCount = bs, count
        }
    }
    return best, best != nil
}

The one detail worth calling out: the scan starts from a rotating offset, the same counter round robin uses, instead of always starting at index 0. When several backends are tied at the same in-flight count (common when load is light, or every backend happens to be equally fast), always picking the first one found would quietly turn "least connections" back into "whichever backend happens to be first in the list," undoing the whole point. Rotating the starting point breaks ties fairly across backends instead.

The benchmark

Two backends, deliberately uneven: one answering instantly, one with an artificial 50 millisecond delay added to every response. Fifty concurrent clients, three seconds, same setup, same load, strategy as the only variable:

Least-connections pushed 48% more total throughput through the exact same two backends. That's not a rounding-error difference, and it's not subtle: round robin kept sending half of all traffic to the slow backend no matter what, because it has no way to notice a backend is slow in the first place. Least-connections noticed the slow backend still had requests in flight and routed around it, automatically, without anyone telling it which backend was the slow one.

The part I didn't expect

The p95 and p99 numbers are very slightly worse for least-connections, 65ms and 70ms versus round robin's 60ms and 65ms. My first instinct was that this had to be noise. It isn't, and the reason it isn't is worth sitting with: least-connections balances load, it does not avoid a backend just for being slower. The slow backend still gets picked, whenever its in-flight count happens to be the lowest at that instant, which under real concurrent load still happens reasonably often. Those requests are still in the sample, and they're still slow, so they still show up in the tail. What least-connections buys you is fewer of them overall, not zero of them, and the tail reflects that honestly rather than hiding it.

That's the actual lesson, more than the throughput number: these algorithms don't eliminate a slow backend's effect on your system, they decide how much of that effect you absorb versus route around. A benchmark that only reported the throughput win and left this part out would have been telling a cleaner story than the one that's actually true.

Why keep both

Nothing about RelayForge needed two strategies to function, round robin alone would have been a complete, working load balancer. I kept least-connections specifically so the two could be benchmarked against each other once there was an actual reason to, uneven backend cost, rather than building a pluggable strategy interface speculatively and hoping it would be useful someday. It's the same reasoning I used building a storage engine before this: don't generalize until you have a concrete case that needs it. Here, the concrete case was "what happens when backends aren't all the same speed," and the answer turned out to be a 48% throughput difference I wouldn't have trusted myself to guess at correctly without measuring it.

Full implementation, including the tie-breaking detail, is on GitHub.