Feature request: work offline

[flisky]

background

Gopkg.toml & Gopkg.lock are committed into git.
$GOPATH/pkg/dep is cached between CIs.

What did you expect to see?

There is a way to restore the exact version from Gopkg.lock, and
dep only fetches the remote when $GOPATH/pkg/dep doesn't have the version.

This will speed up the CI process, especially for GFW in China.

What did you see instead?

This is no way to work offline.

[sdboyer]

Yeah, offline mode is something we'd like to add. It's complicated to get it right, though, so - unless someone steps up and is willing to champion a rather difficult project 😉 - it's not likely we'll get to this anytime soon.

However!

The particular case you're describing doesn't actually require an explicit offline mode. It only requires that, if we have the necessary data in the local cache, dep doesn't reach out to the internet. That's much more feasible. I'd even describe it as optimal, and the obvious thing we should be aiming for. It would still be a bit tricky and involved, but if there's someone willing to do some spelunking, I I'm happy to guide them through it.

[flisky]

The doc says: If the dependency is missing from vendor/, the latest version allowed by your manifest will be installed.

I'd like to change dep ensure to restore the locked version if Gopkg.lock has, and make latest version dedicated to dep ensure -update. It's more intuitable to me.

If a teammate commits the version, the other one could pull a newer-but-regression version under current behavior. dep ensure is supposed to run often, and the default behavior should go following the lock file.

Any thoughts, @sdboyer ?

[sdboyer]

I'd like to change dep ensure to restore the locked version if Gopkg.lock has, and make latest version dedicated to dep ensure -update. It's more intuitable to me.

ugh. that's how it works now. that bit of the README is wrong 😢

[niondir]

I face the following usecase pretty regular:

ProjectB depends on ProjectA

I edit projectA and then like to update the vendored dependency of projectB

Currently the only way is to do a pretty slow dep ensure -v -update projectA

I see few options how to solve this:

• Make dep smart, and minimize the time to at maximum one remote request to see that projectA is already up to date. (Probably the most complicated)
• Let dep ensure -v projectA copy over the local version of projectA to projectB (does not happen currently, not sure if I can add something to my Gopkg.toml to achive this)
• Introduce dep ensure -v -update -offline projectA to skip all potential remote calls (might be simple to implement but maybe outdated in future)

[sdboyer]

@niondir ah, so that's an orthogonal case than the offline workflow - i tend to refer to it as the "multi-project workflow." some discussion in #935.

[tq-schifferm]

For the sake of using dep as a fetcher in an OpenEmbedded recipe, I would be interested in the following:

• No network access when Gopkg.lock is up-to-date and all referenced commits are found in the cache
• -offline flag to explicitly prohibit all network access, making dep fail when a dependency is not cached (@sdboyer: You describe an explicit offline mode as less feasible than implicitly working offline; I don't really understand the reasoning though. Shouldn't it be easy to cut off all network access, simply failing when a dependency is not available?)

At the moment, the first failure when trying to dep ensure without network access is the metadata fetch in gps/deduce.go:

"golang.org/x/crypto/bcrypt": unable to deduce repository and source type for "golang.org/x/crypto/bcrypt": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://golang.org/x/crypto/bcrypt?go-get=1"

Is there already a concept how to handle this without network access? Is the data we have in the cache sufficient to do without the metadata? I've seen a few unmerged PRs on that subject, but I don't think any provided a conclusive solution so far.

At a glance, it seems that at least the mapping from dependency names to cache directory names depends on the metadata, so I assume we'd need to do one of:

1. base package cache directories on package names rather than source URL
2. cache metadata as well
3. store actual source URL in Gopkg.lock

If there are no other reasons we need to get the metadata, 1. sounds like the best solution to me.

[Poseiden]

@schiffermtq Yes, I met the same problem that us production-build network can't access metadata(of golang.org/x/crypto lib), and no workaround. So we decide to find other build tool that support offline. So sorry about that. @sdboyer

[niondir]

All my issues are gone with go mod now. This issue is one of the major issues why I switched.

[sdboyer]

not really clear how the module system improves on that, but sure

[niondir]

It at least speeds things up a lot while I don't need new versions from remote repositories by caching more versions.