cmd/compile/internal/escape: performance issue

[podtserkovskiy]

Go version

go version go1.22.3 linux/amd64

Output of go env in your module/workspace:

go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/michaelpo/.cache/go-build'
GOENV='/home/michaelpo/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/michaelpo/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/michaelpo/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/michaelpo/go/1.22.3/linux_amd64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/home/michaelpo/go/1.22.3/linux_amd64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.3'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/michaelpo/code/go_escape_bug/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build204849496=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Repo with full repro and pprof profiles: https://github.com/podtserkovskiy/go-slow-escape

We have a Go package generated from a large thrift file Bad.thrift by github.com/facebook/fbthrift codegen.
The Bad.thrift file has ~12k significant lines of code and generated Go package has ~924k significant lines of code.
It takes 43 minutes for cmd/compile to compile it, which seems to be reasonable until we start to compare it with other files.

We have another big thrift file Good.thrift, it has ~11k significant lines of code and generated Go package has ~737k significant lines of code.
However it takes just 4 minutes to compile this Go package.

The key difference between these files is that Bad.thrift structures is more interconnected than Good.thrift.

The compilation time of Bad.thrift expected to be a bit longer than Good.thrift, but 39 min difference (10x!) looks suspicious.

What did you see happen?

My next step after discovering this facts was adding -cpuprofile flag to cmd/compile and the results look very interesting.

The compiler spends ~74% of time in cmd/compile/internal/escape.Funcs when processing Bad.thrift.
However Good.thrift's profile looks completely different, it spends majority of time in ssagen.Compile instead.

SVG files for both Bad.thrift svg and Good.thrift svg are attached.

The raw pprof profile files are there good.cpu and bad.cpu.

What did you expect to see?

That makes me think we hit a performance bottleneck there.
I would be really grateful if somebody familiar with this part of the compiler can take a look on this issue 🙏

[mauri870]

cc @golang/compiler

[thanm]

Thanks for the report.

For enormous generated packages of this sort (800k lines, 95k functions) you are virtually guaranteed to run into compile time disasters of one sort or another; if not in escape analysis, then in some other portion of the back end.

Go is an open source project, if you'd like to debug the problem and perhaps send a patch, you're welcome to go ahead with that, but at least from my perspective working on these sorts of issues doesn't seem like a great use of time for the C&R team.

The underlying problem here is that the code generator in question is manufacturing a single enormous monolithic package, instead of breaking its output up into chunks or smaller units. This approach simply doesn't scale when inputs get larger and larger.

Have you filed an issue with the project that owns the code generator?

[podtserkovskiy]

Thanks for the reply.

Have you filed an issue with the project that owns the code generator?
Yes, I'm in touch with them. The maintainers are working on the output minimisation.

I totally understand that this is not an average case. I decided to report the issue, because we have two same-sized huge packages with very different escape analysis time.
That made me think this issue may be interested for someone.

We can close this issue, if that's not the case.

[podtserkovskiy]

We're working on redesigning github.com/facebook/fbthrift to avoid this issue.
I'm closing this issue since nobody else have this problem 😀