cmd/link: linking github.com/hashicorp/packer takes >2gb of ram

[lyda]

What version of Go are you using (go version)?

Using the version from ports:

$ go version
go version go1.12.5 freebsd/amd64

Does this issue reproduce with the latest release?

I think that's the latest release, so yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOOS="freebsd"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/freebsd_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build861311780=/tmp/go-build -gno-record-gcc-switches"

What did you do?

As mentioned in #27739, building packer on a t2.small with around 1.5g of ram will fail. A full terraform example is here and will reproduce the problem.

What did you expect to see?

I expected packer to be linked.

What did you see instead?

The Go compiler crashed at the link stage. If you follow the terraform example and upgrade to t2.medium, you'll see it finally succeed - it will have 3.8g of ram available (check with vmstat).

[odeke-em]

Hello @lyda thank you for reporting this issue and welcome to the Go project!

I'll page some experts about this @ianlancetaylor @randall77 @josharian @aclements, but am not sure if we should triage this for Go1.13 or Go1.14?

[lyda]

Thanks. And no worries about versions, whichever version makes sense. I was using the one users of FreeBSD are likely to use.

Note, I provided a simple terraform config to make it easier to reproduce. Finding the right FreeBSD AMI and navigating AWS if you don't know it is a pain. So the terraform config is just there to make it easier for you to get a test machine to try it on.

You can get the same outcome on a similarly sized non-AWS VM, I just happened to be setting up an AWS instance at the time. The base issue is memory usage in linking. The packer project seems to present an example body of code that takes what seems like an excessive amount of ram to link.

I think the changed title is a bit misleading because the issue really isn't with terraform. It's merely a convenience I provided to make people's lives easier in reproducing the issue. And I only included packer in the title because that's what I was compiling - I suspect packer is hardly the only code base where this issue occurs but I have not tried compiling a bunch of Go projects to prove that.

[odeke-em]

Thank you @lyda! I've updated it to "cmd/compile: out of memory on t2.small when linking/building with less than 1.5GB of RAM", please feel free to modify it. You have access too to update the title :)

[josharian]

I tried doing GO111MODULE=off /usr/bin/time -l go build -a github.com/hashicorp/packer with Go 1.11, 1.12, and tip. The maxrss was 2.25gb, 2.49gb, and 2.09gb respectively. So 1.13 is moving in the right direction.

cc @thanm and @cherrymui who have been working on linker memory reductions

I'm going to tentatively mark this as 1.13, in case either of them sees something easy and safe to do here, inspired particularly by linking packer. It is highly likely this will get punted to 1.14, though.

[lyda]

2.13 looks promising - just need to shave off .6g or so! :)

[cherrymui]

Quick question: Does it use cgo (i.e. internal linking or external linking)? Do you build a static executable, or PIE/shared library/etc. which requires dynamic relocations? Also, how big are the inputs (object files) and the output? Thanks!

[lyda]

Not really sure.

# file /usr/local/bin/packer
/usr/local/bin/packer: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, Go BuildID=F_YMeGLIsqFoRNLB81mG/2EtlqnuY6GqeXjDDVPun/WiejIm7lSAh-xMMbUR98/kcs28g1e9gfaaebfZlUt, stripped

It's made with this Makefile. It includes a lot, but I think go.mk is the relevant one.

[ALTree]

It looks like the recent linker improvements fixed this issue.

I tried building github.com/hashicorp/packer with tip (soon to be go1.15) on an aws t2.small instance, and it worked.

$ ec2-metadata -t
instance-type: t2.small

$ grep MemTotal /proc/meminfo
MemTotal:        2039152 kB

$ ~/gotip/bin/go version
go version devel +11f92e9 Mon Jul 20 18:18:56 2020 +0000 linux/amd64

$ /usr/bin/time -v ~/gotip/bin/go build -a github.com/hashicorp/packer
	User time (seconds): 309.87
	System time (seconds): 33.28
	Percent of CPU this job got: 95%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 5:58.23
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 1757880
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 4988
	Minor (reclaiming a frame) page faults: 8098532
	Voluntary context switches: 242554
	Involuntary context switches: 162508
	Swaps: 0
	File system inputs: 2104784
	File system outputs: 2374984
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

I'm closing this issue, but feel free to comment if you believe it should be kept open.

[aclements]

Thanks for closing the loop on this!