net/http: transport MaxIdleConnsPerHost does not work as expected

[sschepens]

I recently had an issue which made Go consume all ephemeral ports and start failing to do requests.
The problem was triggered because I was doing 10 concurrent requests with the DefaultMaxIdleConnsPerHost so Transport was only reusing 2 connections and closing the other 8 as soon as a request was finished, just to reestablish a connection again.
This seems a really bad behavior and never happened to me on any other language or http client.
A connection is usually considered idle when it was not used for a period of inactivity, maybe a whole keepalive period or so, not after a request has been sent successfully.
The behavior one would expect is to have 10 established connections and, after say 30 seconds of not issuing any request having only 2 connections established as per DefaultMaxIdleConnsPerHost.

Here is a gist that triggers this behavior.

I'm running go 1.5.2 on Ubuntu Precise 12.04, kernel: Linux e-0000e768 3.8.0-44-generic #66~precise1-Ubuntu SMP Tue Jul 15 04:01:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[ianlancetaylor]

CC @bradfitz

[bradfitz]

This has been like this forever, so it's not Go 1.6 material. It's also somewhat involved.

[gopherbot]

CL https://golang.org/cl/18282 mentions this issue.

[bradfitz]

Related, probably to be addressed at the same time: #13957 (limiting number of dials in-flight at once per host)

[bradfitz]

Related also: we should close idle conns in some defined order, like the oldest use one gets close first.

[bradfitz]

Related also: we should close idle conns in some defined order, like the oldest use one gets close first.

That happened in Go 1.7 at least.

The rest of this bug didn't happen, though.