Why Connection Pooling Matters More Than It Sounds Like It Should

"Reuse connections instead of opening a new one every time" sounds like exactly the kind of optimization you'd file under micro, the sort of thing that shaves a few percent off a benchmark nobody outside a systems team will ever look at. While building RelayForge, a reverse proxy in Go, I decided not to take that on faith. I built a second, deliberately bad version with no pooling at all, and benchmarked them against each other. The gap wasn't a few percent.

What dialing fresh actually costs

Every TCP connection starts with a three-way handshake: SYN, SYN-ACK, ACK, three full round trips between the two sides before a single byte of your actual request has gone anywhere. On a fast network that's genuinely fast, a millisecond or two. The problem isn't really the handshake's raw latency, it's that opening and closing sockets has a cost on both sides of the connection that compounds under load in a way a single request's timing never shows you.

A proxy that pools connections pays that handshake cost once per backend connection, then reuses it for however many requests come through while it stays open. A proxy that doesn't pays it on literally every single request, forever, no matter how many requests are flowing through:

// the naive version: a fresh dial, every request, no exceptions
backend, err := net.Dial("tcp", backendAddr)
if err != nil {
    return // no retry, no pool, nothing to fall back on
}
defer backend.Close() // always closed afterward, every time

Versus the pooled version, which checks for an already-open, idle connection first and only dials when it doesn't have one:

backend := s.takeIdleConn(backendAddr)
if backend == nil {
    backend, err = net.DialTimeout("tcp", backendAddr, s.connectTimeout())
    if err != nil {
        return attempt{err: err}
    }
}
// ...after a clean response, instead of closing:
s.putIdleConn(backendAddr, backend)

takeIdleConn and putIdleConn are a small map of backend address to a slice of idle connections, guarded by a mutex, capped at a handful per backend so it can't grow without bound. That's genuinely most of what pooling is: a place to put a connection down when you're done with it, and a place to check before deciding you need a new one.

The benchmark that convinced me

I built naive-proxy, a second, intentionally simple proxy with the exact same HTTP-aware forwarding logic as RelayForge but none of the optimizations: fresh dial every request, no health checks, no retries. Then I pointed a load generator at both, same backend, same concurrency, same duration, and measured.

At 100 concurrent clients:

At 200 concurrent clients, the gap got worse, not better:

Four times the throughput, and the naive version wasn't just slower, it was actively failing more than a third of its requests by 200 concurrent clients. I didn't expect that second part going in. I expected pooling to win on speed. I didn't expect the unpooled version to start outright breaking.

Why it was actually failing, not just slow

The failures weren't backend errors or timeouts. Checking what was actually going wrong, every one of them was the same OS-level error: WSAEADDRINUSE, "only one usage of each socket address is normally permitted." That's what you get when you try to open a TCP connection and the local ephemeral port you'd use is still tied up in TIME_WAIT from a connection that was just closed moments ago.

Every connection close doesn't instantly free up its port for reuse. The OS holds it in TIME_WAIT for a while (on Windows, by default, around two minutes) specifically so a very late, stray packet from the old connection can't get misdelivered into a brand new one that happens to reuse the same four-tuple too soon. That's a real, necessary safety mechanism. It's also a real, finite resource: there are only so many ephemeral ports available, and a proxy that opens and closes a fresh backend connection on every single request is burning through that pool constantly, every closed connection sitting in TIME_WAIT for two minutes before it's usable again.

Under light load, you'd never notice. Under 200 concurrent clients hammering the same proxy, the naive version was generating new connections faster than the OS could recycle old ones out of TIME_WAIT, and started running out of local ports to use. That's not a performance problem you tune your way out of with a faster backend or a beefier machine, it's a structural consequence of the dial-per-request design, and it gets worse the more successful your proxy is at attracting traffic.

The pooled version never touches this problem at all for sustained traffic, because it isn't constantly opening and closing backend connections in the first place. I confirmed this directly: after firing five separate client requests at the pooled proxy, netstat showed exactly one established connection to the backend the entire time, five client connections, one backend connection, reused five times.

The honest caveat

A pooled connection can still go stale between being returned to the pool and being picked up again, the backend is always free to close an idle connection on its own schedule, and the pool has no way to know until it actually tries to use it. RelayForge handles this by treating a failed write to a reused connection as a strong signal the connection, not the request, was the problem, and retrying once against a fresh one. That's a separate piece of the design, worth its own explanation, but it only exists because pooling exists. You don't need a policy for stale pooled connections if you never pool connections in the first place, you just pay the dial cost again and again, and apparently, past a certain point, pay it in outright failures too.

The full implementation, including the idle pool and the retry logic that covers for it, is on GitHub.