net: 1.14 performance regression on mac

[erikdubbelboer]

What version of Go are you using (go version)?

$ go version
go version go1.13.5 darwin/amd64

and

$ go version
go version go1.14beta1 darwin/amd64

Does this issue reproduce with the latest release?

Yes, the issue started with the latest beta release.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/erik/Library/Caches/go-build"
GOENV="/Users/erik/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/erik"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/s6/xkrsszq51hb0vrnmn08jsr140000gn/T/go-build119700800=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I wrote a simple program that uses the net package to use 9000 connections and 9000*2 goroutines (*2 for both sides of the connection) to send requests and responses every 200 milliseconds. Then I measure the average delay it takes for the response to arrive (so a little bit over 200ms expected).
Program: https://gist.github.com/erikdubbelboer/d925c6e271aa0c5c8bb20a6ec16455c5

What did you expect to see?

On go1.13.5 the average response time is around 206ms. So 6ms more than the expected delay.

What did you see instead?

On go1.14beta1 the average response time is 300ms. That's 94ms extra delay just for upgrading Go.

The program only consumes about half of the CPU's I have so it's not a CPU bottleneck.

I have tested the exact same thing on an Ubuntu machine and here the average response time is 200ms (rounded down, it's only a couple of microseconds of delay) for both versions of Go.

[agnivade]

Thank you for this report @erikdubbelboer. Unfortunately, I don't have a mac, so I can't test this. Would you be open to running a git bisect to exactly pin point this issue to the offending commit ? That would help a lot in the investigation.

[erikdubbelboer]

Some information I found bisecting.

go1.13rc1 had bad performance for my program. Which was fixed in 81d995d (runtime: fix darwin syscall performance regression).

c1635ad (runtime: fix darwin syscall performance regression) fixed it for master at the same time. But it went back to bad performance again with 6becb03 (runtime: switch to using new timer code). But not nearly as bad as it was before the syscall fix.

[josharian]

cc @ianlancetaylor

[ianlancetaylor]

I'm seeing similar behavior with Go 1.12, Go 1.13, and tip. The program starts out reporting a latency of about 200ms. As the connections get up to the full 9000, the latency starts to increase. For me on macOS 10.15 it seems to settle around 220ms to 250ms. This is an example of what I see. Are these the kinds of numbers you see as well? Do you really see a difference between 1.13 and tip? I'm not seeing it. I see similar delays on GNU/Linux on my laptop.

connections:  343  latency: 200.890396ms  (requests: 683)
connections:  697  latency: 200.821368ms  (requests: 2418)
connections:  1061  latency: 200.743184ms  (requests: 4192)
connections:  1434  latency: 200.733952ms  (requests: 6017)
connections:  1808  latency: 200.761542ms  (requests: 7896)
connections:  2187  latency: 200.714418ms  (requests: 9797)
connections:  2583  latency: 200.639253ms  (requests: 11672)
connections:  2992  latency: 200.642091ms  (requests: 13695)
connections:  3404  latency: 200.817111ms  (requests: 15702)
connections:  3828  latency: 200.752635ms  (requests: 17853)
connections:  4268  latency:  200.64982ms  (requests: 19982)
connections:  4715  latency: 200.606723ms  (requests: 22118)
connections:  5165  latency: 200.740386ms  (requests: 24346)
connections:  5615  latency: 200.706656ms  (requests: 26657)
connections:  6072  latency: 201.176292ms  (requests: 28991)
connections:  6545  latency: 201.548536ms  (requests: 30761)
connections:  7024  latency: 201.627506ms  (requests: 33446)
connections:  7503  latency: 202.091374ms  (requests: 35604)
connections:  7997  latency: 212.458301ms  (requests: 36428)
connections:  8467  latency: 216.730454ms  (requests: 36998)
connections:  8929  latency: 227.487405ms  (requests: 38116)
connections:  9000  latency: 251.811791ms  (requests: 35618)
connections:  9000  latency: 240.457705ms  (requests: 37968)
connections:  9000  latency: 237.856099ms  (requests: 37371)
connections:  9000  latency: 234.247637ms  (requests: 38453)
connections:  9000  latency: 256.271508ms  (requests: 34680)
connections:  9000  latency: 247.740709ms  (requests: 36829)
connections:  9000  latency: 236.892206ms  (requests: 37733)
connections:  9000  latency: 236.814958ms  (requests: 38422)
connections:  9000  latency: 226.538981ms  (requests: 39630)
connections:  9000  latency: 239.949913ms  (requests: 37197)
connections:  9000  latency: 226.853039ms  (requests: 40724)
connections:  9000  latency: 226.292862ms  (requests: 39254)
connections:  9000  latency: 229.269346ms  (requests: 39063)
connections:  9000  latency: 235.468175ms  (requests: 37769)
connections:  9000  latency: 220.767293ms  (requests: 41690)
connections:  9000  latency: 224.779376ms  (requests: 39430)
connections:  9000  latency: 256.439405ms  (requests: 34553)
connections:  9000  latency: 229.501098ms  (requests: 39812)
connections:  9000  latency: 242.980294ms  (requests: 37187)
connections:  9000  latency: 234.840228ms  (requests: 38104)
connections:  9000  latency: 216.742172ms  (requests: 41804)
connections:  9000  latency: 232.979317ms  (requests: 38800)
connections:  9000  latency: 222.800823ms  (requests: 40336)
connections:  9000  latency: 233.061475ms  (requests: 38445)
connections:  9000  latency: 246.251366ms  (requests: 36510)
connections:  9000  latency: 244.446013ms  (requests: 36494)
connections:  9000  latency: 247.226202ms  (requests: 37439)
connections:  9000  latency: 221.866441ms  (requests: 39705)
connections:  9000  latency: 227.286424ms  (requests: 39899)
connections:  9000  latency: 223.551262ms  (requests: 40033)
connections:  9000  latency:  245.03208ms  (requests: 36543)

[ianlancetaylor]

I think part of the problem is that the timer implementation tends to cause all your servers to synchronize. They all write data at the same time and the readers all run into each other, so some see higher latency.

If I change the handler function as follows, the numbers are slightly better, but the program still gets overwhelmed on macOS 10.15 (it seems stable at 200ms on my laptop).

func handler(c net.Conn) {
	buff := make([]byte, 4)
	for {
		if _, err := c.Read(buff); err != nil {
			panic(err)
		}
		off := (rand.Int() % 16) - 8
		time.Sleep(responseDelay + time.Duration(off)*time.Millisecond)
		if _, err := c.Write(buff); err != nil {
			panic(err)
		}
	}
}

[ianlancetaylor]

Any program like this is bound to collapse at some level of connections. As I said above, I'm seeing similar results with Go 1.13 and tip. Can you confirm that you see better results with 1.13? If you push up the number of connections, it will collapse on 1.13; what number of connection causes that collapse? Thanks.

[erikdubbelboer]

I have a 2019 Macbook pro (2,3 GHz 8-Core) with 10.15.2.

This is a run on tip (go version devel +dcd3b2c173 Fri Dec 27 14:52:12 2019 +0000 darwin/amd641):

connections:  214  latency: 200.602515ms  (requests: 420)
connections:  433  latency: 200.708629ms  (requests: 1501)
connections:  654  latency: 200.724819ms  (requests: 2592)
connections:  878  latency: 200.721209ms  (requests: 3707)
connections:  1104  latency: 200.697312ms  (requests: 4811)
connections:  1333  latency: 200.653961ms  (requests: 5972)
connections:  1566  latency: 200.545604ms  (requests: 7107)
connections:  1799  latency: 200.676203ms  (requests: 8293)
connections:  2033  latency: 200.810307ms  (requests: 9441)
connections:  2267  latency: 200.825205ms  (requests: 10541)
connections:  2502  latency: 200.596215ms  (requests: 11805)
connections:  2739  latency:  200.62905ms  (requests: 12957)
connections:  2978  latency: 200.517847ms  (requests: 14150)
connections:  3217  latency: 200.539781ms  (requests: 15216)
connections:  3458  latency: 200.700861ms  (requests: 16513)
connections:  3698  latency: 200.447265ms  (requests: 17752)
connections:  3937  latency: 200.696562ms  (requests: 18862)
connections:  4176  latency:  200.63284ms  (requests: 20144)
connections:  4418  latency: 200.346688ms  (requests: 21362)
connections:  4662  latency:  200.69621ms  (requests: 22556)
connections:  4905  latency: 200.712916ms  (requests: 23775)
connections:  5148  latency: 202.988265ms  (requests: 24897)
connections:  5391  latency: 201.089358ms  (requests: 26045)
connections:  5634  latency: 200.549375ms  (requests: 27386)
connections:  5877  latency: 200.697466ms  (requests: 28579)
connections:  6122  latency: 205.651507ms  (requests: 28230)
connections:  6366  latency:   209.0161ms  (requests: 29643)
connections:  6609  latency: 203.697572ms  (requests: 32314)
connections:  6851  latency: 206.174856ms  (requests: 32867)
connections:  7094  latency: 206.177448ms  (requests: 32674)
connections:  7336  latency: 223.981978ms  (requests: 32597)
connections:  7507  latency: 305.926898ms  (requests: 25405)
connections:  7724  latency: 305.048324ms  (requests: 23950)
connections:  7912  latency: 296.864946ms  (requests: 27612)
connections:  8093  latency: 337.062981ms  (requests: 18546)
connections:  8297  latency: 322.061387ms  (requests: 26544)
connections:  8495  latency: 316.352134ms  (requests: 27000)
connections:  8681  latency: 315.528667ms  (requests: 28212)
connections:  8882  latency:  305.82877ms  (requests: 30176)
connections:  9000  latency: 330.977453ms  (requests: 22154)
connections:  9000  latency: 362.380999ms  (requests: 27621)
connections:  9000  latency: 332.453846ms  (requests: 28781)
connections:  9000  latency: 320.721338ms  (requests: 28836)
connections:  9000  latency: 285.075168ms  (requests: 31100)
connections:  9000  latency: 251.145905ms  (requests: 33885)
connections:  9000  latency: 266.993671ms  (requests: 34766)
connections:  9000  latency: 314.165259ms  (requests: 26445)
connections:  9000  latency: 345.336108ms  (requests: 25636)
connections:  9000  latency: 361.087745ms  (requests: 28983)
connections:  9000  latency: 295.295363ms  (requests: 30191)
connections:  9000  latency: 339.091424ms  (requests: 22256)
connections:  9000  latency: 293.209347ms  (requests: 37219)
connections:  9000  latency: 286.418834ms  (requests: 31369)
connections:  9000  latency: 338.615434ms  (requests: 25998)
connections:  9000  latency: 302.197456ms  (requests: 26307)
connections:  9000  latency:  339.63387ms  (requests: 27489)
connections:  9000  latency: 345.729196ms  (requests: 28294)
connections:  9000  latency: 296.134887ms  (requests: 29979)

While this is a run on go version go1.13.5 darwin/amd64:

connections:  206  latency: 201.149826ms  (requests: 406)
connections:  416  latency: 200.932811ms  (requests: 1445)
connections:  628  latency: 200.991975ms  (requests: 2493)
connections:  844  latency: 200.945477ms  (requests: 3545)
connections:  1061  latency: 200.918099ms  (requests: 4639)
connections:  1280  latency: 201.072663ms  (requests: 5734)
connections:  1498  latency: 201.003851ms  (requests: 6771)
connections:  1719  latency: 201.163361ms  (requests: 7895)
connections:  1942  latency: 201.064027ms  (requests: 9034)
connections:  2165  latency: 201.100604ms  (requests: 10112)
connections:  2387  latency: 201.302735ms  (requests: 11237)
connections:  2613  latency: 201.103637ms  (requests: 12310)
connections:  2837  latency: 201.162047ms  (requests: 13473)
connections:  3060  latency: 201.149551ms  (requests: 14530)
connections:  3283  latency: 201.063125ms  (requests: 15556)
connections:  3510  latency: 201.138935ms  (requests: 16813)
connections:  3739  latency: 201.106901ms  (requests: 17954)
connections:  3969  latency: 200.785103ms  (requests: 19103)
connections:  4198  latency: 200.967839ms  (requests: 20288)
connections:  4424  latency: 201.233829ms  (requests: 21369)
connections:  4655  latency: 201.029016ms  (requests: 22482)
connections:  4886  latency: 200.937855ms  (requests: 23499)
connections:  5099  latency: 237.045212ms  (requests: 21638)
connections:  5307  latency: 201.191379ms  (requests: 25906)
connections:  5520  latency: 201.021086ms  (requests: 26960)
connections:  5732  latency: 201.196859ms  (requests: 28005)
connections:  5947  latency: 201.223842ms  (requests: 28388)
connections:  6156  latency: 201.125707ms  (requests: 30039)
connections:  6368  latency: 201.053725ms  (requests: 31157)
connections:  6580  latency: 201.053327ms  (requests: 31922)
connections:  6792  latency: 201.039457ms  (requests: 32863)
connections:  7005  latency: 201.009148ms  (requests: 34001)
connections:  7218  latency: 201.153349ms  (requests: 35231)
connections:  7430  latency: 201.104808ms  (requests: 36403)
connections:  7642  latency: 201.181692ms  (requests: 37396)
connections:  7854  latency: 201.059726ms  (requests: 38416)
connections:  8066  latency: 201.171463ms  (requests: 39493)
connections:  8279  latency: 201.136226ms  (requests: 40496)
connections:  8491  latency: 201.199482ms  (requests: 41736)
connections:  8700  latency: 201.219195ms  (requests: 42761)
connections:  8911  latency: 201.325808ms  (requests: 43509)
connections:  9000  latency:  201.12872ms  (requests: 44478)
connections:  9000  latency: 201.443376ms  (requests: 44568)
connections:  9000  latency: 202.122442ms  (requests: 44627)
connections:  9000  latency: 203.712159ms  (requests: 44141)
connections:  9000  latency: 224.089598ms  (requests: 38300)
connections:  9000  latency: 205.978398ms  (requests: 44455)
connections:  9000  latency: 202.635591ms  (requests: 45000)
connections:  9000  latency: 204.245141ms  (requests: 45000)
connections:  9000  latency: 205.165565ms  (requests: 44985)
connections:  9000  latency: 203.643746ms  (requests: 44517)
connections:  9000  latency:  205.35368ms  (requests: 45000)
connections:  9000  latency: 207.054406ms  (requests: 42367)
connections:  9000  latency: 204.450071ms  (requests: 42891)
connections:  9000  latency: 207.518396ms  (requests: 42052)
connections:  9000  latency: 205.640631ms  (requests: 44188)
connections:  9000  latency: 205.211299ms  (requests: 45000)
connections:  9000  latency: 205.542529ms  (requests: 44677)
connections:  9000  latency: 221.424168ms  (requests: 37812)
connections:  9000  latency: 216.833935ms  (requests: 44775)
connections:  9000  latency: 203.923357ms  (requests: 44981)
connections:  9000  latency: 204.407349ms  (requests: 44999)
connections:  9000  latency: 203.890018ms  (requests: 43897)
connections:  9000  latency: 205.134139ms  (requests: 42448)
connections:  9000  latency:   202.2458ms  (requests: 44625)

As you can see go1.13.5 is quite a lot better on my machine.

[erikdubbelboer]

@ianlancetaylor what minor version of 1.13 did you use? 1.13.2 was the first version where it was fixed.

With 1.13.5 the collapsing seems to start happening around 16000 connections on my machine:

connections:  10081  latency: 201.543594ms  (requests: 49629)
connections:  10287  latency: 201.842059ms  (requests: 50637)
connections:  10492  latency: 201.619908ms  (requests: 51731)
connections:  10697  latency: 201.800619ms  (requests: 52655)
connections:  10902  latency: 202.044265ms  (requests: 53543)
connections:  11109  latency: 201.760951ms  (requests: 54709)
connections:  11315  latency:  201.67879ms  (requests: 55913)
connections:  11520  latency: 201.629559ms  (requests: 55798)
connections:  11728  latency: 201.725815ms  (requests: 57648)
connections:  11932  latency: 201.624005ms  (requests: 58427)
connections:  12137  latency: 201.747662ms  (requests: 59554)
connections:  12340  latency:   201.7525ms  (requests: 60301)
connections:  12544  latency: 201.881979ms  (requests: 61589)
connections:  12747  latency: 202.253278ms  (requests: 62605)
connections:  12951  latency: 201.842114ms  (requests: 63771)
connections:  13156  latency: 203.120681ms  (requests: 64749)
connections:  13356  latency: 202.875871ms  (requests: 64827)
connections:  13557  latency: 202.099093ms  (requests: 66139)
connections:  13760  latency: 202.912646ms  (requests: 67198)
connections:  13962  latency:  203.19292ms  (requests: 68694)
connections:  14166  latency: 202.143645ms  (requests: 69493)
connections:  14367  latency: 205.881382ms  (requests: 69463)
connections:  14568  latency: 202.632688ms  (requests: 71619)
connections:  14767  latency: 202.851463ms  (requests: 71808)
connections:  14965  latency: 202.405885ms  (requests: 73269)
connections:  15166  latency: 203.179725ms  (requests: 74557)
connections:  15369  latency: 202.990612ms  (requests: 75057)
connections:  15569  latency:  203.49432ms  (requests: 76009)
connections:  15770  latency: 204.251799ms  (requests: 76570)
connections:  15968  latency: 208.079449ms  (requests: 76338)
connections:  16166  latency: 212.495423ms  (requests: 74899)
connections:  16367  latency: 219.372552ms  (requests: 73830)
connections:  16564  latency: 225.440395ms  (requests: 72965)
connections:  16768  latency: 233.959613ms  (requests: 71311)
connections:  16965  latency: 238.820195ms  (requests: 70405)
connections:  17163  latency:   239.0021ms  (requests: 71058)
connections:  17364  latency: 243.902386ms  (requests: 70551)
connections:  17564  latency: 247.059713ms  (requests: 70397)
connections:  17767  latency: 253.104628ms  (requests: 69966)
connections:  17969  latency: 249.847045ms  (requests: 71142)
connections:  18171  latency: 264.383372ms  (requests: 68061)
connections:  18370  latency: 266.974333ms  (requests: 68307)
connections:  18567  latency: 283.147913ms  (requests: 64881)
connections:  18768  latency: 269.575891ms  (requests: 69489)
connections:  18963  latency: 273.408313ms  (requests: 68427)
connections:  19000  latency: 274.476879ms  (requests: 69898)
connections:  19000  latency: 279.551007ms  (requests: 68027)
connections:  19000  latency: 269.565461ms  (requests: 70508)
connections:  19000  latency: 272.486302ms  (requests: 68795)
connections:  19000  latency:  274.05602ms  (requests: 70395)
connections:  19000  latency: 270.531462ms  (requests: 69769)
connections:  19000  latency: 273.705709ms  (requests: 69718)

[ianlancetaylor]

I was testing with the tip of the 1.13 branch, which is a couple of CLs past 1.13.5.

What happens on your machine if you tweak the handler function as I suggested above?