pull-cluster-api-verify is broken after the upgrade to go 1.17 rc1 CI image

[sbueringer]

What steps did you take and what happened:
[A clear and concise description on how to REPRODUCE the bug.]

Currently, all our pull-cluster-api-verify are failing. Our jobs were updated yesterday (link) to use the new kubekins-e2e:v20210715-521d667-go-canary image. The job fails now because go 1.17 rc1 adds additional //go:build lines at the top of our generated files (details).

What did you expect to happen:
Shouldn't break that way.

Anything else you would like to add:
This affects all providers on all branches that:

• use the go-canary image in CI
• generate code and diff for changes in CI

Possible solutions:

Change the CI images to one that is using Go 1.16
This should be possible by moving from the kubekins-e2e:v20210715-521d667-go-canary to the kubekins-e2e:v20210715-521d667-master image (which currently uses go 1.16.6). Assumption is that it's reasonable to assume the latest stable Go version on dev machines and align to that in CI.

Stop using the local Go version for code generation
Run our generate targets with a specified go version and not the local one (at least in CI). Probably the easiest way to do this is to run it in a container. For example: docker run -it -v $(PWD):/go/src/sigs.k8s.io/cluster-api -w /go/src/sigs.k8s.io/cluster-api --entrypoint /bin/bash --privileged gcr.io/k8s-testimages/kubekins-e2e:v20210715-521d667-go-canary runner.sh ./scripts/ci-verify.sh.

Add a post-processing step which adds the header
The post-processing could be implemented via e.g.:

• a custom go tool (shouldn't be very complicated)
• find / sed (but in a way that works on Linux and Mac) (Credits to @randomvariable Slack)
• running gofmt 1.17

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[randomvariable]

The issue with "running gofmt 1.17", is we don't yet know of an easy way to make this automatic via some magic in hack/tools, such that local development isn't blocked on using gvm or something.

[vincepri]

We should switch the images for Cluster API to v1.21, which is still using Go 1.16 and it's our main dependency in Controller Runtime

[vincepri]

/milestone v0.4