x/tools/go/gcexportdata: can't import package from pkg/mod

[xushiwei]

What version of Go are you using (go version)?

$ go version
go version go1.18 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/xushiwei/Library/Caches/go-build"
GOENV="/Users/xushiwei/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/xushiwei/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/xushiwei/go"
GOPRIVATE=""
GOPROXY=""
GOROOT="/Users/xushiwei/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/xushiwei/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/xushiwei/work/gox/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yb/cjx6c2ks4tn7pdmj9s5zz3ph0000gn/T/go-build3409594572=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

https://go.dev/play/p/1Us1Xqxqori

package main

import (
	"fmt"
	"go/token"
	"go/types"
	"os"
	"path/filepath"
	"runtime"

	"golang.org/x/tools/go/gcexportdata"
)

func main() {
	val := filepath.Join(runtime.GOROOT(), "pkg/mod")
	os.Setenv("GOMODCACHE", val)
	fmt.Println("GOMODCACHE:", val)

	fset := token.NewFileSet()
	packages := make(map[string]*types.Package)
	imp := gcexportdata.NewImporter(fset, packages)

	_, err := imp.Import("go/types")
	fmt.Println("Import result:", err)

	_, err = imp.Import("golang.org/x/tools/go/gcexportdata")
	fmt.Println("Import result:", err)
}

What did you expect to see?

GOMODCACHE: /Users/xushiwei/go/pkg/mod
Import result: <nil>
Import result: <nil>

What did you see instead?

Local output:

GOMODCACHE: /Users/xushiwei/go/pkg/mod
Import result: <nil>
Import result: can't find import: golang.org/x/tools/go/gcexportdata

Output of go.dev/play:

GOMODCACHE: /usr/local/go-faketime/pkg/mod
Import result: can't find import: go/types
Import result: can't find import: golang.org/x/tools/go/gcexportdata

[xushiwei]

package main

import (
	"fmt"
	"go/importer"
	"go/token"
	"go/types"
	"os"
	"path/filepath"
	"runtime"

	"golang.org/x/tools/go/gcexportdata"
)

func main() {
	val := filepath.Join(runtime.GOROOT(), "pkg/mod")
	os.Setenv("GOMODCACHE", val)
	fmt.Println("GOMODCACHE:", val)

	var fset = token.NewFileSet()
	var imp types.Importer
	if true {
		imp = importer.ForCompiler(fset, "source", nil)
	} else {
		packages := make(map[string]*types.Package)
		imp = gcexportdata.NewImporter(fset, packages)
	}

	_, err := imp.Import("go/types")
	fmt.Println("Import result:", err)

	_, err = imp.Import("golang.org/x/tools/go/gcexportdata")
	fmt.Println("Import result:", err)
}

got expected result:

GOMODCACHE: /Users/xushiwei/go/pkg/mod
Import result: <nil>
Import result: <nil>

But it ran slowly (not in this example, but in a real package).

[ianlancetaylor]

CC @golang/tools-team

[findleyr]

As of https://go.dev/cl/310515, gcexportdata.NewImporter will be deprecated, IIUC for the reasons present in this CL. Can you use x/tools/go/packages instead?

CC @rsc @matloob

[xushiwei]

As of https://go.dev/cl/310515, gcexportdata.NewImporter will be deprecated, IIUC for the reasons present in this CL. Can you use x/tools/go/packages instead?

CC @rsc @matloob

I need a types.Importer, but x/tools/go/packages hasn't. And, packages.Load is slow and problematic. If I call packages.Load multiple times, it will create multiple instances of same depended packages. And therefore fails to checking identical types by go/types.

[findleyr]

@xushiwei depending on your use-case, it is probably not too hard to implement an importer by memoizing the response of packages.Load. If you don't ask for source/syntax, this should load from export data. If that is still too slow it is worth profiling to understand why.

I'd be interested in hearing how this goes.

[xushiwei]

@xushiwei depending on your use-case, it is probably not too hard to implement an importer by memoizing the response of packages.Load. If you don't ask for source/syntax, this should load from export data. If that is still too slow it is worth profiling to understand why.

I'd be interested in hearing how this goes.

I had written an importer by packages.Load. There is a lot of trouble here.

For example, suppose package E depends A and B, and package F depends A and C.

When I loaded E, it created an A package instance (called it A1) and B. And when I loaded F, it also create an A package instance (called it A2) and C.

The big trouble is: E depends A1, F depends A2, but types.Identical(A1.Foo, A2.Foo) is NOT true.

A normal importer doesn't have this problem. When I load E, it created an A package instance (called it A1) and B. And when I loaded F, it reused A1 and only create a C package instance.

[findleyr]

I had written an importer by packages.Load. There is a lot of trouble here

Right, sorry I didn't really think through my suggestion.

Next suggestion: use go/packages only to get the correct exportfile (using NeedsExportFile), and use gcexportdata.Read. Example here. Then you can use memoized imports... and I think it should work.

[findleyr]

FWIW, after https://go.dev/cl/310515 it looks gcexportdata.Import should actually work correctly (despite being deprecated), so you could also test at that CL. As the commit message indicates using go/packages is preferable, but if you are only loading one package at a time they are probably equivalent.

[xushiwei]

@findleyr Thx. I will try your suggestion.

[xushiwei]

@findleyr I try

pkgs, err := packages.Load(&packages.Config{Dir: dir, Mode: packages.NeedExportsFile}, "fmt")

and it returns ENOENT.

But I use go list -export directly:

type listExport struct {
	Export string `json:"Export"`
}

func findExport(dir, pkgPath string) (expfile string) {
	var ret listExport
	if data, err := golistExport(dir, pkgPath); err == nil {
		json.Unmarshal(data, &ret)
	}
	return ret.Export
}

func golistExport(dir, pkgPath string) (ret []byte, err error) {
	var stdout, stderr bytes.Buffer
	cmd := exec.Command("go", "list", "-json", "-export", pkgPath)
	cmd.Stdout = &stdout
	cmd.Stderr = &stderr
	cmd.Dir = dir
	err = cmd.Run()
	if err == nil {
		ret = stdout.Bytes()
	}
	return
}

It works well.

[iqpkeq]

@xushiwei

go/types/resolver.go collectObjects() line:259

fileDir := dir(check.fset.Position(file.Name.Pos()).Filename)
check.importPackage(d.spec.Path, path, fileDir)

Has the same problem as you.Maybe not fixed yet.
If you use

check.Files(files)   //types.check 

to check source code. It will cause the same error