cmd/go: toolchain directive in go.mod being updated unnecessarily

[dprotaso]

There doesn't seem to be a way to stop the toolchain directive from being updated when running various go commands

In our OSS project we rely on the go directive in the go.mod to ensure a min go version. We do not want to use the toolchain directive as we always expect it to match the go directive.

some user feedback

• https://gophers.slack.com/archives/C0VPK4Z5E/p1704471476388119
• https://gophers.slack.com/archives/C9BMAAFFB/p1708525315637839

Go Version

go version go1.22.0 darwin/amd64

Go ENV

expand

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/Users/dprotasowski/Library/Caches/go-build'
GOENV='/Users/dprotasowski/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/dprotasowski/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/dprotasowski/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/Cellar/go/1.22.0/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/Cellar/go/1.22.0/libexec/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/dprotasowski/work/test/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/xs/xfp1z4cn643c46w69lv8bv8w0000gp/T/go-build4163821748=/tmp/go-build -gno-record-gcc-switches -fno-common'

Scenario 1 - creating a new module and downgrading go

What did you do

$ mkdir test && cd test
$ go mod init blah
$ go get [email protected]

What did you see happen?

• Tool chain directive of 1.22.0 was added - but this isn't necessary
• min go version set to 1.21.7

What did you expect to happen?

• No toolchain directive to be added
• Min go version set to 1.21 - since I didn't specify the point release

Scenario 2 - pulling a new dependency that upgraded the go version

What did you do

Project structure

--- go.mod
module blah

go 1.19

---- main.go
package main

import "fmt"

func main() {
	fmt.Println("vim-go")
}

Fetch a dependency that updates the go min version

$ go get knative.dev/pkg@9f033a7

What did you see happen?

console output

go: upgraded go 1.19 => 1.21
go: added toolchain go1.22.0
go: added knative.dev/pkg v0.0.0-20240221065059-9f033a7b77f7

go.mod output

module blah

go 1.21

toolchain go1.22.0

require knative.dev/pkg v0.0.0-20240221065059-9f033a7b77f7 // indirect

What did you expect to happen?

toolchain directive shouldn't appear. We want it to always default to the same as the go directive

[bcmills]

Please fill out the complete issue template.

In particular: what version of Go are you using, what was your go directive set to initially, and what commands did you run that added the toolchain directive?

[dprotaso]

@bcmills updated - can you take a look again and see if I'm missing anything?

[dylan-bourque]

Adding my own experiences for reference.

At work, we would definitely prefer to have the ability to disable the automatic toolchain upgrades (and possibly fail builds that would otherwise require a newer toolchain). We have many modules with the go directive in their go.mod files targeting Go versions from 1.16 to 1.21. We also do not proactively update those directives across the codebase when a new Go version is released.

This feature started causing issues for us when one engineer using Go 1.21.x locally updated some dependency and ran go mod tidy, which silently updated the go directive in that module and added the new toolchain one. We were all surprised when this change failed to build in CI, which was still using Go 1.20.x.

Without the ability to opt out of automatic toolchain upgrades we are now forced to synchronize the Go version across CI and every developer machine. While that's not an unreasonable stance in the general case, it is a new restriction/requirement that, as far as I've seen, is not called out in the release notes.

We have tried setting GOTOOLCHAIN to local but that doesn't seem to affect the behavior of go mod tidy. It will still upgrade both the go and toolchain directives.

One could also argue that forcing more conservative development shops to always stay on the most current Go release, despite the previous minor release being explicitly supported, is a bad situation.

[bcmills]

As of Go 1.21, the initial toolchain release has a .0 suffix (see #57631), so go 1.21 is a development version of the language — not a released version.

And for the toolchain directive in particular, per https://go.dev/doc/toolchain: “For repeatability, any command that updates the go line also updates the toolchain line to record its own toolchain name.”

Some specifics:

• go get knative.dev/pkg@9f033a7 is adding the toolchain line because it is upgrading the go line. That seems to be working as designed.

• go get [email protected] is resolving to a specific point release because that's what it usually does for version prefixes. Unfortunately, the language version is a prefix of the release version, and we don't have a good way to force the language version instead of resolving the prefix. That probably needs some more thought.

• I don't understand why go get [email protected] is not writing out a toolchain line. That is at the very least a mismatch between the documentation and the implementation.

[bcmills]

At work, we would definitely prefer to have the ability to disable the automatic toolchain upgrades (and possibly fail builds that would otherwise require a newer toolchain).

You do have that ability: you can set GOTOOLCHAIN=local in your process environment or GOENV file. (You can even edit $GOROOT/go.env to make that the default for your Go toolchain installation, if you are so inclined.)

You can also force go get to stick to a particular go version, the same way you would for any other dependency that you want to avoid upgrading, by passing that version as an explicit argument to go get: go get [email protected] will downgrade to go 1.20, and go get knative.dev/pkg@9f033a7 [email protected] will correctly error out (and report that those versions are not mutually compatible).

Without the ability to opt out of automatic toolchain upgrades we are now forced to synchronize the Go version across CI and every developer machine.

That is exactly the problem that GOTOOLCHAIN=auto is supposed to mitigate: when everyone is on Go 1.21.0 or above, the developers' machines should automatically download whatever toolchain is needed to work in the module.

[dylan-bourque]

You do have that ability: you can set GOTOOLCHAIN=local in your process environment

This works, but it also means that as soon as anyone moves to a newer Go release then everyone else in the organization must also move to that same newer version immediately. This is a problem in tightly regulated environments where, as an example Go 1.22.x may not be "certified" but Go 1.21.5 is. As it stands today it's not possible to both fix CI to 1.21.x and also allow developers to use a newer 1.21.y or 1.22.

You can also force go get to stick to a particular go version ... by passing that version as an explicit argument to go get: go get [email protected]

TIL. Thanks for this.

[cevich]

you can set GOTOOLCHAIN=local in your process environment

There are environments where setting an env. var. is not possible. For example, the public Renovate service does not support this. There are other ways to pin the golang version in Renovate, but they're undesirable for maintainability reasons. Devcontainers could be another example, where the tooling env. is shared, and must remain static, for consistency across a whole team of developers. In any case, my point is there are places where setting env. vars. isn't a viable workaround. IMHO such major behavior changes should almost always default to the previous or "least disruptive" option.

[gaby]

I just ran into this same issue, go1.21 is supported but my box has go1.22. I can't run any commands because golang keeps forcing the toolchain directive into the go.mod file.

This is for a library used by a lot of users and it should support go1.21 and go1.22

[XANi]

Still a problem, it just makes a complete mess any time common CI server is used by few people

[dmitshur]

I'm not seeing this already mentioned here, but in situations where you're looking to change the go directive and also not have a toolchain directive, have you considered explicitly including toolchain@none in the go get command? For example:

$ cat go.mod
module example.com

go 1.21.0

$ go get [email protected] toolchain@none
go: upgraded go 1.21.0 => 1.22.6

That comes with the disadvantages of not having the toolchain version written down, but that's no different from the outcome where it's manually removed after go get adds it.

[prymitive]

As of Go 1.21, the initial toolchain release has a .0 suffix (see #57631), so go 1.21 is a development version of the language — not a released version.

A while ago one would set the go field in go.mod to specify the language version the project requires at minimum. The change to use full go release as the value of go, rather than just the major/minor release numbers as before, means that every single module might specify a different point release when in reality all patch releases of Go X.Y are guaranteed to be compatible - and if they are not that would be for some very good reason and is rare.
So every library out there putting a different point release in their go.mod combined with this automatic bumping of go / toolchain means there's a lot of noise in PR diffs that does not (in my opinion) adds any value at all. It's just churn.

And for the toolchain directive in particular, per https://go.dev/doc/toolchain: “For repeatability, any command that updates the go line also updates the toolchain line to record its own toolchain name.”

Some specifics:

• go get knative.dev/pkg@9f033a7 is adding the toolchain line because it is upgrading the go line. That seems to be working as designed.
• go get [email protected] is resolving to a specific point release because that's what it usually does for version prefixes. Unfortunately, the language version is a prefix of the release version, and we don't have a good way to force the language version instead of resolving the prefix. That probably needs some more thought.

A while ago go.mod would be the files that's maintained / validated by the code author to write down requirements and constrains of the project. The go.sum file would be the lock file that holds all the low level details of how to resolve this constrains / requirements.
With this automatic bumping of go and insertion/updates of toolchain go.mod starts to also be a mini-lock file that encodes some of that information, which I think is unfortunate.
The decision for all of that happens automatically, unless the use puts in place enough env variables or files that stops go command from doing all of this means that there's by default a magic happening where it wasn't before, and maybe all of that will get easier and more obvious over time, but at present (for me personally) it took away simplicity of go.mod away while not giving me anything in return.

[dnwe]

I'm not seeing this already mentioned here, but in situations where you're looking to change the go directive and also not have a toolchain directive, have you considered explicitly including toolchain@none in the go get command?

Thanks for posting this. That's exactly the behaviour I (and probably others) had been looking for and was originally the behaviour that I thought that GOTOOLCHAIN=local would provide for me — i.e., always use the toolchain I have installed locally and never add a toolchain directive in the go.mod file. I always have the most recent release kept up-to-date locally, but I don't want that to keep getting stamped into the go.mod when its unnecessary.

Certainly for my own individual modules I'd likely move to a model of always doing:
go get toolchain@none go@{{- .GoLangVersion }}.0 <otherpkg> [otherpkgs...] to retain the behaviour of the go directive declaring the minimum language and stdlib version required to build the module, and then I'm happy for whatever toolchain >= to that version to be used at build time.

Ideally I'd want to bake that client behaviour into the go.mod file so that when anyone else updates a dependency, it retains the go directive and doesn't add an arbitrary toolchain directive based on whatever version they happened to have installed locally.

[matthewhartstonge]

I would argue that by adding toolchain as a non opt-in feature breaks user expectations around Go's release policy due to shadow forcing people on to whatever latest version is injected into go.mod.

Release Policy¶
Each major Go release is supported until there are two newer major releases. For example, Go 1.5 was supported until the Go 1.7 release, and Go 1.6 was supported until the Go 1.8 release. We fix critical problems, including critical security problems, in supported releases as needed by issuing minor revisions (for example, Go 1.6.1, Go 1.6.2, and so on).

I have found this really painful for library developers as well. I've had to ask maintainers to lower the versioning or remove the toolchain directive in a popular OSS lib to support said release policy.

As with others, I'm finding I'm constantly wasting time having to unwind/remove the toolchain directive as we run a n-1 Go versioning policy at our place of work.

[XANi]

It's so bad I'm considering adding git filter for it. Every time anyone runs even minor release difference compared to build server it breaks

[gopherbot]

Change https://go.dev/cl/656835 mentions this issue: cmd/go: don't always write toolchain line when updating go line

[matloob]

I think this behavior is ultimately confusing and I don't know of any users that are depending on the reproducibility benefits of it. So we're going to stop adding a toolchain line for the local toolchain when the go line is increased.

That said, I would like to hear from anyone depending on the current behavior.

Thanks for all your comments and feedback!

[gopherbot]

Change https://go.dev/cl/657178 mentions this issue: doc/next: add release note for new toolchain line behavior