proposal: cmd/go: add -reproducible flag

[tie]

There seems to be a growing number of flags required to get reproducible builds with Go (e.g. see #51748). It’d be nice if there was -reproducible flag that would

• Disable CGO_ENABLED and GO_EXTLINK_ENABLED by default, unless explicitly set via environment variables.
• Default to -reproducible=true for go install. Currently there are some platforms (e.g. android) where Cgo is required, but those are usually cross-compiled so CGO_ENABLED must be set explicitly anyway.
• Forbid setting -buildvcs=auto explicitly—there is no reason to set that if the goal is reproducible builds.
• Default to -trimpath=true and -buildvcs=true flags, unless explicitly set via command line arguments or GOFLAGS environment variable.

Additionally, for -reproducible=true and -buildvcs=true, given a non-dirty VCS source tree of an example.com module, go build -reproducible should produce the same output as go install example.com@version-or-revision (even if the current revision was not pushed yet). Otherwise, if the tree is dirty, build should fail.

If go.mod contains replace and other directives that would cause go install to fail, go build should fail too with -reproducible=true.

Using VCS information to make go build reproducible against go install seems to be non-trivial to implement, so alternatively perhaps we can just set -buildvcs=false by default with -reproducible flag.

[ianlancetaylor]

I think it could be confusing for a cmd/go flag like -reproducible to disable cgo. Many people want reproducible builds and many program use cgo. Having -reproducible disable cgo would be a confusing experience.

[tie]

Setting up reproducible builds with C toolchain is already an explicit step since that relies heavily on the state of the host system (e.g. presence of the system libraries), and I don’t think an extra step of setting an environment variable would be confusing for people that need reproducible builds with Cgo. Also, currently Cgo must be enabled explicitly when cross-compiling.

That said, on a second thought, that would only cause an issue with go install that under this proposal would have -reproducible=true (and hence Cgo disabled), and I’d like to retract this part.

This way, builds that have to be reproducible can run with GOFLAGS=-reproducible without changing any existing behavior in Go toolchain when the flag is not set.

[seankhliao]

I don't really see a point in having a flag that can have every setting it controls then be overridden by something else, it makes it harder to reason about which setting is actually taking effect, especially combined with setting flags in GOFLAGS.
And having a flag named like this but only getting you half of the way there (if you use CGo) seems worse than having no such flag to mislead you.

[rsc]

We're not going to make a -reproducible flag disable cgo. That's mixing two unrelated things.

At that point I don't see the difference between -reproducible and -trimpath.

[rsc]

There are some other reproducibility issues we found with recording things like CGO_FLAGS in the binaries. I think we're planning to key off -trimpath to fix them. So probably -trimpath is the reproducibility flag.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[nightlyone]

Is it possible to instead document that intention of -trimpath meaning "reproducible build"? That would also allow people to file bugs that way, if code fails to build from source.

[beoran]

If -trimpath is the -reproducible flag, then the latter is a better name for it. We could likely deprecate the former and replace it with the latter.

[rsc]

/cc @bcmills for thoughts on renaming/aliasing/deprecating -trimpath to -reproducible?

[bcmills]

I think -trimpath is the more appropriate word for what the flag does today.

Notably, it does not audit or stamp the versions of external C headers or libraries linked into the resulting binary, so using the same Go release to build a cgo-enabled binary with -trimpath on two different machines (or on one machine with intervening changes to system C libraries) does not necessarily reproduce the exact binary.

I think a -reproducible flag conceptually ought to error out if any of the other flags might result in a non-reproducible binary: so, for example, rather than disabling cgo implicitly, -reproducible might error out completely. But then I'm not sure how useful it would really be.

[mvdan]

I agree with Bryan; Go has enough misconceptions as it stands today, and I can certainly see this one appearing in no time:

use the -reproducible flag to make your builds reproducible

It is a requirement to make reproducible builds possible, but it's not enough in many scenarios, and I imagine it will rarely be enough in most large projects.

[rsc]

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

[mvdan]

I think that a better way to solve the problem of "how to make Go builds reproducible" would be documentation. Like others said, making your build reproducible will depend on your program and how you want to build it.

I already filed #48540 some time ago, and it overlaps with reproducible builds already, so I think we can use it as an umbrella issue. Essentially, we want the docs to cover everything that one needs to consider to make builds reproducible, such as:

• use the -trimpath flag
• ensure that the cgo build is reproducible (toolchain, dependencies, etc), or disable cgo
• ensure that the version of Go used is recorded
• ensure that all build parameters (-tags, GOOS, GOAMD64, etc) are recorded

[FiloSottile]

I agree with @mvdan. The other piece of the puzzle that I think would be extremely valuable would be a debug mode that prints every input to the build ID. I'm in a thread on Twitter where developers were complaining about a "random" build ID causing Arch Linux reproducibility failures, and it would be useful to tell them "run it under -debug-build-id and check the diff" instead of a guessing game of what is causing the issue.

[Foxboron]

ensure that the cgo build is reproducible (toolchain, dependencies, etc), or disable cgo

But is there any testing on this part? Currently it seems like cgo is not tested and the ability to test them seems hard at best. Building delve under two chroots with CGO and the same flags produce different buildids which gets propagated to the buildid of the binary.

CGO is desirable because of binary hardening which is lacking in the main Go compiler.

[Foxboron]

Also having things like GODEBUG=gocachehash=1 better documented would be nice I think for issues like this.

[ianlancetaylor]

The other piece of the puzzle that I think would be extremely valuable would be a debug mode that prints every input to the build ID.

I won't claim that the output is easy to read, but you get this by setting GODEBUG=gocachehash=1.

[Foxboron]

The above mentioned issue from twitter is LTO symbols having random numbers in their suffix. I tried adding -frandom-seed=0 a few places in the compiler but I haven't managed to get them deterministic :/

We should maybe document the current cases where Go is not reproducible?

Diffoscope output:

    λ something » diffoscope _pkg_.a repro.a
    --- _pkg_.a
    +++ repro.a
    ├── file list
    │ @@ -1,16 +1,16 @@
    │  ?rw-r--r--   0        0        0     2036 1970-01-01 00:00:00.000000 __.PKGDEF
    │  ?rw-r--r--   0        0        0    36887 1970-01-01 00:00:00.000000 _go_.o
    │  ?rw-r--r--   0        0        0     1096 1970-01-01 00:00:00.000000 asm_amd64.o
    │  ?rw-r--r--   0        0        0     2784 1970-01-01 00:00:00.000000 _x001.o
    │  ?rw-r--r--   0        0        0     2776 1970-01-01 00:00:00.000000 _x002.o
    │  ?rw-r--r--   0        0        0     4536 1970-01-01 00:00:00.000000 _x003.o
    │ -?rw-r--r--   0        0        0     7208 1970-01-01 00:00:00.000000 _x004.o
    │ +?rw-r--r--   0        0        0     7224 1970-01-01 00:00:00.000000 _x004.o
    │  ?rw-r--r--   0        0        0    14504 1970-01-01 00:00:00.000000 _x005.o
    │  ?rw-r--r--   0        0        0    10176 1970-01-01 00:00:00.000000 _x006.o
    │  ?rw-r--r--   0        0        0     6680 1970-01-01 00:00:00.000000 _x007.o
    │  ?rw-r--r--   0        0        0     4936 1970-01-01 00:00:00.000000 _x008.o
    │ -?rw-r--r--   0        0        0     8544 1970-01-01 00:00:00.000000 _x009.o
    │ +?rw-r--r--   0        0        0     8528 1970-01-01 00:00:00.000000 _x009.o
    │  ?rw-r--r--   0        0        0     4720 1970-01-01 00:00:00.000000 _x010.o
    │  ?rw-r--r--   0        0        0     7752 1970-01-01 00:00:00.000000 _x011.o
    │  ?rw-r--r--   0        0        0    16568 1970-01-01 00:00:00.000000 _x012.o
    │  ?rw-r--r--   0        0        0      864 1970-01-01 00:00:00.000000 _x013.o
    ├── __.PKGDEF
    │ @@ -1,16 +1,16 @@
    │  00000000: 676f 206f 626a 6563 7420 6c69 6e75 7820  go object linux
    │  00000010: 616d 6436 3420 676f 312e 3138 2e33 2058  amd64 go1.18.3 X
    │  00000020: 3a72 6567 6162 6977 7261 7070 6572 732c  :regabiwrappers,
    │  00000030: 7265 6761 6269 7265 666c 6563 742c 7265  regabireflect,re
    │  00000040: 6761 6269 6172 6773 2c70 6163 6572 7265  gabiargs,pacerre
    │  00000050: 6465 7369 676e 0a62 7569 6c64 2069 6420  design.build id
    │  00000060: 224b 325f 3375 582d 6e6e 4d59 657a 6748  "K2_3uX-nnMYezgH
    │ -00000070: 7677 3630 762f 484e 5645 6779 2d37 7a66  vw60v/HNVEgy-7zf
    │ -00000080: 6637 7162 5842 796b 6170 220a 0a0a 2424  f7qbXBykap"...$$
    │ +00000070: 7677 3630 762f 4539 4f4d 7252 7763 3236  vw60v/E9OMrRwc26
    │ +00000080: 7679 746f 4249 504f 3450 220a 0a0a 2424  vytoBIPO4P"...$$
    │  00000090: 420a 6902 8705 9309 2124 474f 524f 4f54  B.i.....!$GOROOT
    │  000000a0: 2f73 7263 2f72 756e 7469 6d65 2f63 676f  /src/runtime/cgo
    │  000000b0: 2f68 616e 646c 652e 676f 0556 616c 7565  /handle.go.Value
    │  000000c0: 0168 0648 616e 646c 6500 0644 656c 6574  .h.Handle..Delet
    │  000000d0: 6503 7e72 3001 7602 6f6b 034d 6170 0473  e.~r0.v.ok.Map.s
    │  000000e0: 796e 6304 4c6f 6164 0768 616e 646c 6573  ync.Load.handles
    │  000000f0: 1724 474f 524f 4f54 2f73 7263 2f73 796e  .$GOROOT/src/syn
    ├── _go_.o
    │ @@ -1,16 +1,16 @@
    │  00000000: 676f 206f 626a 6563 7420 6c69 6e75 7820  go object linux
    │  00000010: 616d 6436 3420 676f 312e 3138 2e33 2058  amd64 go1.18.3 X
    │  00000020: 3a72 6567 6162 6977 7261 7070 6572 732c  :regabiwrappers,
    │  00000030: 7265 6761 6269 7265 666c 6563 742c 7265  regabireflect,re
    │  00000040: 6761 6269 6172 6773 2c70 6163 6572 7265  gabiargs,pacerre
    │  00000050: 6465 7369 676e 0a62 7569 6c64 2069 6420  design.build id
    │  00000060: 224b 325f 3375 582d 6e6e 4d59 657a 6748  "K2_3uX-nnMYezgH
    │ -00000070: 7677 3630 762f 484e 5645 6779 2d37 7a66  vw60v/HNVEgy-7zf
    │ -00000080: 6637 7162 5842 796b 6170 220a 0a0a 2424  f7qbXBykap"...$$
    │ +00000070: 7677 3630 762f 4539 4f4d 7252 7763 3236  vw60v/E9OMrRwc26
    │ +00000080: 7679 746f 4249 504f 3450 220a 0a0a 2424  vytoBIPO4P"...$$
    │  00000090: 0a0a 2424 0a0a 0a24 2420 202f 2f20 6367  ..$$...$$  // cg
    │  000000a0: 6f0a 5b5b 2263 676f 5f65 7870 6f72 745f  o.[["cgo_export_
    │  000000b0: 7374 6174 6963 222c 2263 726f 7373 6361  static","crossca
    │  000000c0: 6c6c 3222 5d2c 5b22 6367 6f5f 6578 706f  ll2"],["cgo_expo
    │  000000d0: 7274 5f64 796e 616d 6963 222c 2263 726f  rt_dynamic","cro
    │  000000e0: 7373 6361 6c6c 3222 5d2c 5b22 6367 6f5f  sscall2"],["cgo_
    │  000000f0: 6578 706f 7274 5f73 7461 7469 6322 2c22  export_static","
    ├── _x001.o
    │ ├── readelf --wide --sections {}
    │ │ @@ -2,27 +2,27 @@
    │ │
    │ │  Section Headers:
    │ │    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
    │ │    [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
    │ │    [ 1] .text             PROGBITS        0000000000000000 000040 000000 00  AX  0   0  1
    │ │    [ 2] .data             PROGBITS        0000000000000000 000040 000000 00  WA  0   0  1
    │ │    [ 3] .bss              NOBITS          0000000000000000 000040 000000 00  WA  0   0  1
    │ │ -  [ 4] .gnu.lto_.profile.f64d60f5ec95f2ee PROGBITS        0000000000000000 000040 00000f 00   E  0   0  1
    │ │ -  [ 5] .gnu.lto_.icf.f64d60f5ec95f2ee PROGBITS        0000000000000000 00004f 000011 00   E  0   0  1
    │ │ -  [ 6] .gnu.lto_.ipa_sra.f64d60f5ec95f2ee PROGBITS        0000000000000000 000060 000011 00   E  0   0  1
    │ │ -  [ 7] .gnu.lto_.inline.f64d60f5ec95f2ee PROGBITS        0000000000000000 000071 000011 00   E  0   0  1
    │ │ -  [ 8] .gnu.lto_.jmpfuncs.f64d60f5ec95f2ee PROGBITS        0000000000000000 000082 000011 00   E  0   0  1
    │ │ -  [ 9] .gnu.lto_.pureconst.f64d60f5ec95f2ee PROGBITS        0000000000000000 000093 00000e 00   E  0   0  1
    │ │ -  [10] .gnu.lto_.ipa_modref.f64d60f5ec95f2ee PROGBITS        0000000000000000 0000a1 000011 00   E  0   0  1
    │ │ -  [11] .gnu.lto_.lto.f64d60f5ec95f2ee PROGBITS        0000000000000000 0000b2 000008 00   E  0   0  1
    │ │ -  [12] .gnu.lto_.symbol_nodes.f64d60f5ec95f2ee PROGBITS        0000000000000000 0000ba 00000f 00   E  0   0  1
    │ │ -  [13] .gnu.lto_.refs.f64d60f5ec95f2ee PROGBITS        0000000000000000 0000c9 00000e 00   E  0   0  1
    │ │ -  [14] .gnu.lto_.decls.f64d60f5ec95f2ee PROGBITS        0000000000000000 0000d7 000026 00   E  0   0  1
    │ │ -  [15] .gnu.lto_.symtab.f64d60f5ec95f2ee PROGBITS        0000000000000000 0000fd 000000 00   E  0   0  1
    │ │ -  [16] .gnu.lto_.ext_symtab.f64d60f5ec95f2ee PROGBITS        0000000000000000 0000fd 000001 00   E  0   0  1
    │ │ +  [ 4] .gnu.lto_.profile.fd3e93c4e939fc22 PROGBITS        0000000000000000 000040 00000f 00   E  0   0  1
    │ │ +  [ 5] .gnu.lto_.icf.fd3e93c4e939fc22 PROGBITS        0000000000000000 00004f 000011 00   E  0   0  1
    │ │ +  [ 6] .gnu.lto_.ipa_sra.fd3e93c4e939fc22 PROGBITS        0000000000000000 000060 000011 00   E  0   0  1
    │ │ +  [ 7] .gnu.lto_.inline.fd3e93c4e939fc22 PROGBITS        0000000000000000 000071 000011 00   E  0   0  1
    │ │ +  [ 8] .gnu.lto_.jmpfuncs.fd3e93c4e939fc22 PROGBITS        0000000000000000 000082 000011 00   E  0   0  1
    │ │ +  [ 9] .gnu.lto_.pureconst.fd3e93c4e939fc22 PROGBITS        0000000000000000 000093 00000e 00   E  0   0  1
    │ │ +  [10] .gnu.lto_.ipa_modref.fd3e93c4e939fc22 PROGBITS        0000000000000000 0000a1 000011 00   E  0   0  1
    │ │ +  [11] .gnu.lto_.lto.fd3e93c4e939fc22 PROGBITS        0000000000000000 0000b2 000008 00   E  0   0  1
    │ │ +  [12] .gnu.lto_.symbol_nodes.fd3e93c4e939fc22 PROGBITS        0000000000000000 0000ba 00000f 00   E  0   0  1
    │ │ +  [13] .gnu.lto_.refs.fd3e93c4e939fc22 PROGBITS        0000000000000000 0000c9 00000e 00   E  0   0  1
    │ │ +  [14] .gnu.lto_.decls.fd3e93c4e939fc22 PROGBITS        0000000000000000 0000d7 000026 00   E  0   0  1
    │ │ +  [15] .gnu.lto_.symtab.fd3e93c4e939fc22 PROGBITS        0000000000000000 0000fd 000000 00   E  0   0  1
    │ │ +  [16] .gnu.lto_.ext_symtab.fd3e93c4e939fc22 PROGBITS        0000000000000000 0000fd 000001 00   E  0   0  1
    │ │    [17] .gnu.lto_.opts    PROGBITS        0000000000000000 0000fe 0000e5 00   E  0   0  1
    │ │    [18] .comment          PROGBITS        0000000000000000 0001e3 000013 01  MS  0   0  1
    │ │    [19] .note.GNU-stack   PROGBITS        0000000000000000 0001f6 000000 00      0   0  1
    │ │    [20] .note.gnu.property NOTE            0000000000000000 0001f8 000050 00   A  0   0  8
    │ │    [21] .symtab           SYMTAB          0000000000000000 000248 000048 18     22   2  8
    │ │    [22] .strtab           STRTAB          0000000000000000 000290 00001e 00      0   0  1
    │ │    [23] .shstrtab         STRTAB          0000000000000000 0002ae 00022d 00      0   0  1
    

[Foxboron]

And @seankhliao pointed out the proper flag is -frandom-seed=1. Submitted the change at #53528 but I probably don't have a complete overview of what it entails.

[rsc]

It's fine to work on a doc (maybe a wiki page to start?) explaining the subtleties that go into a reproducible build. It's clear that -reproducible isn't trivial to do right now, so we should hold off on that step. (It certainly shouldn't disable cgo, for example.)

[rsc]

No change in consensus, so declined.
— rsc for the proposal review group

[mvdan]

It's fine to work on a doc (maybe a wiki page to start?) explaining the subtleties that go into a reproducible build.

This proposal is now closed, so I've opened #53976 to track that.