dep ensure fails when working directory includes a symlink

[ajessup]

When running dep ensure for the first time in the root of a project cloned from git, I get a cryptic response:
ensure Prepare: at least one package must be present in the PackageTree

This is with a fresh installation of Go (1.7.5) and dep. The project is in my GOPATH.

My manifest.json file is:

{
    "dependencies": {
        "github.com/golang/protobuf": {
            "branch": "master"
        },
        "golang.org/x/net": {
            "branch": "master"
        },
        "google.golang.org/grpc": {
            "branch": "master"
        }
    }
}

My lock.json file is:

{
    "memo": "76828034af0593d3bf68294321e87a2cb63417777e7693726d55d58c1a5ffe61",
    "projects": [
        {
            "name": "github.com/golang/protobuf",
            "branch": "master",
            "revision": "8ee79997227bf9b34611aee7946ae64735e6fd93",
            "packages": [
                "proto"
            ]
        },
        {
            "name": "golang.org/x/net",
            "branch": "master",
            "revision": "9773060888fba93b172cedcd70127db1ab739bd1",
            "packages": [
                "context",
                "http2",
                "http2/hpack",
                "trace"
            ]
        },
        {
            "name": "google.golang.org/grpc",
            "branch": "master",
            "revision": "708a7f9f3283aa2d4f6132d287d78683babe55c8",
            "packages": [
                "."
            ]
        }
    ]
}

[sdboyer]

Odd. Obvious question, but there's at least one directory with *.go files in it in that repo, right?

Is the project you're cloning public, so that I can attempt to replicate?

[ajessup]

There's a couple of subdirectories that have *.go files within them. Unfortunately the repo isn't public, but it's also not especially sensitive. I've attached an archive of the repo that reproduces the bug.
dep-bug-208.tar.gz

[sdboyer]

Seems to work fine for me when I place it at $GOPATH/src/github.com/GoogleCloudPlatform/k8s-render-demo (which I inferred was the expected spot, based on the internal imports) and run dep ensure. Not sure what's going on.

[ajessup]

Out of interest, what version of Go were you testing with?

[sdboyer]

That vm had 1.7.3, looks like

[ajessup]

Curiously, removing the manifest.json and lock.json files and running

$ dep init
$ dep ensure -update

yields the same error

[sdboyer]

And the dir root is at $GOPATH/src/github.com/GoogleCloudPlatform/k8s-render-demo, right?

[ajessup]

Yes

…

On Thu, Feb 9, 2017 at 5:46 AM sam boyer ***@***.***> wrote: And the dir root is at $GOPATH/src/ github.com/GoogleCloudPlatform/k8s-render-demo, right? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#218 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARzY3F3TzsF6koLO6BGVnVHBm4A_6yLks5raxiegaJpZM4L1YNe> .

[sdboyer]

Well, this is really odd. There's gotta be some bit of context/environment that I'm missing here, but I'm not sure what it is.

While I'm waiting for an idea to come to me, all I can really think to do is ask you to instrument the code a bit and report what you see. So, if you could insert a fmt.Println(p, params) (or, better, a pretty.Println(p, params), using github.com/kr/pretty) right before this line, then paste the output into here, that'd be helpful.

[viridicist]

My working directory is $GOPATH/src/foo.com/bar/baz, where baz is a symlink pointing outside of the GOPATH, and I received this error. Swapping the symlink for an actual copy of the code made the error go away. I'm not sure if Go generally supports this symlinky setup, but I believe this is the first time I've run into problems with it.

[ajessup]

Ah, it turns out I had the same issue, and avoiding the symlink fixed the problem.

[sdboyer]

If the case here is supporting the project root being a symlink to elsewhere, then I believe #247 should fix that.

[viridicist]

fwiw, #247 doesn't support my situation where the symlink points outside $GOPATH

[sdboyer]

ughhh symlinks. Yeah, and #247 is stalled a bit right now.

We need a clear, comprehensive plan for this :/ While I've experienced versions of this pain myself plenty of times, I'm really wary about introducing too much symlink magickery without an understanding of the consequences.

[sdboyer]

I wrote up a first stab at a detailed plan for handling symlinks in #247. Should cover this case, I think, though I'm also pretty sure it's got some weak spots as-written.

[sdboyer]

@ajessup Now that we've merged in #247, I'm curious if your original setup works.

@viridicist did the place that #247 ended up cover your case? It should work if your cwd is within the GOPATH, even if the symlink points outside of it (as long as it doesn't point into a different GOPATH).

[viridicist]

@sdboyer With cwd = <my gopath>/src/<my package path>, dep status says

resolve project root: '<my gopath>/src/<my package path>' is linked to another path within a GOPATH (<my gopath>)

So I suppose it resolves the symlink, decides I am not in my gopath, and then complains about my code being in my gopath.

[sdboyer]

@viridicist Hmm...that message should only be given if the root directory of the project is a symlink pointing either to another directory within the current GOPATH, OR a directory within a different GOPATH.

So it doesn't think you're outside your GOPATH - it's saying it won't work because both the tail and head of the symlink are inside a GOPATH.

Looking at my last reply, I think I wasn't clear about the intra-GOPATH links also being excluded - sorry!. The fuller writeup of the mechanics we planned for is here: #247 (comment)

[viridicist]

@sdboyer It's not clear to me how it is coming to that conclusion. Either I really confused dep, or the other way around. 😉

Here is my attempt at reproducing this message:

export GOPATH=$PWD/mygopath
mkdir -p $GOPATH/src
mkdir mypkg
ln -s $PWD/mypkg $GOPATH/src
cd $GOPATH/src/mypkg
dep init
dep status

I believe that is almost verbatim the case 2 in the detailed writeup, but it fails with:

resolve project root: '<$PWD>/mygopath/src/mypkg' is linked to another path within a GOPATH (<$PWD>/mygopath)

[viridicist]

Looking closer, I have a dep from over a week ago installed, let me revisit with current master.

[sdboyer]

@viridicist yeah, i thought you were describing case 2. maybe we have a bug 😄

[viridicist]

So I believe the following should work from any starting point but it fails at dep init with "prepare solver: at least one package must be present in the PackageTree", which I guess is how I got here in the first place. Now I'm not sure if I'm running into what I was trying to reproduce or just using dep wrong, after all, sorry for the confusion.

export GOPATH=$PWD/mygopath
export PATH=$GOPATH/bin:$PATH

go get -u github.com/golang/dep/...

mkdir -p $GOPATH/src/github.com/viridicist
mkdir mypkg
cat > mypkg/foo.go <<END
package foo

import (
  _ "github.com/pkg/errors"
)

func main() {}
END
ln -s $PWD/mypkg $GOPATH/src/github.com/viridicist
cd $GOPATH/src/github.com/viridicist/mypkg
dep init
dep status

[sdboyer]

@viridicist unless i'm missing something, that really does seem like it's the case we intend to cover. do you maybe have time to review our tests for this, see what we're missing, and maybe put up a PR with a fix+improved tests? 🙏 😄

[ajessup]

Just finally got around to re-testing (sorry for the delay) using dep from HEAD. Unfortunately I'm also still seeing the same error prepare solver: at least one package must be present in the PackageTree

[sdboyer]

@ajessup no worries, good to get confirmation, thank you!

prepare solver: at least one package must be present in the PackageTree

It's super the wrong place to be getting the error, but this does suggest to me that what's probably happening is ListPackages() is being fed the symlink path, which ends up not being walked at all (because filepath.Walk()), so you end up with an empty package tree.

Forget what I was saying earlier about the test - it's likely right (I'd context switched out, and when I came back to this I forgot about the new error). The correct fix here is probably to transform the root symlink into the referent dir (assuming it passes our validation checks), while still preserving the inferred ProjectRoot.