Dependency resolution is broken with 'yarn --pnp' (Yarn Plug 'n' Play)

[samsieber]

Bug Report or Feature Request (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request

Command (mark with an x)

- [ ] new
- [x] build
- [ ] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Versions

ng: 6.0.0
yarn: 1.12.0 (RC)
node: v8.11.3
os: OSX 10.13.6

Repro steps

1. Install yarn RC 1.12.0
2. clone https://github.com/samsieber/ng-demo
3. go in the cloned repo
4. run yarn --pnp (switches it to pnp mode)
5. run yarn build

The log given by the failure

Could not find module "@angular-devkit/build-angular" from "/Users/ssieber/dev/ng-yarn-pnp".
Error: Could not find module "@angular-devkit/build-angular" from "/Users/ssieber/dev/ng-yarn-pnp".
    at Object.resolve (/Users/ssieber/Library/Caches/Yarn/v3/npm-@angular-devkit-core-0.6.8-3b09d97bd2588f0091df11921f7ed772431806aa/node_modules/@angular-devkit/core/node/resolve.js:141:11)
    at Observable.rxjs_1.Observable [as _subscribe] (/Users/ssieber/Library/Caches/Yarn/v3/npm-@angular-devkit-architect-0.6.8-977acc605aba45d21b95ca704cc99492e14299dd/node_modules/@angular-devkit/architect/src/architect.js:132:40)
    at Observable._trySubscribe (/Users/ssieber/Library/Caches/Yarn/v3/npm-rxjs-6.3.3-3c6a7fa420e844a81390fb1158a9ec614f4bad55/node_modules/rxjs/internal/Observable.js:44:25)
    at Observable.subscribe (/Users/ssieber/Library/Caches/Yarn/v3/npm-rxjs-6.3.3-3c6a7fa420e844a81390fb1158a9ec614f4bad55/node_modules/rxjs/internal/Observable.js:30:22)
    at DoOperator.call (/Users/ssieber/Library/Caches/Yarn/v3/npm-rxjs-6.3.3-3c6a7fa420e844a81390fb1158a9ec614f4bad55/node_modules/rxjs/internal/operators/tap.js:32:23)
    at Observable.subscribe (/Users/ssieber/Library/Caches/Yarn/v3/npm-rxjs-6.3.3-3c6a7fa420e844a81390fb1158a9ec614f4bad55/node_modules/rxjs/internal/Observable.js:25:22)
    at /Users/ssieber/Library/Caches/Yarn/v3/npm-rxjs-6.3.3-3c6a7fa420e844a81390fb1158a9ec614f4bad55/node_modules/rxjs/internal/util/subscribeTo.js:22:31
    at Object.subscribeToResult (/Users/ssieber/Library/Caches/Yarn/v3/npm-rxjs-6.3.3-3c6a7fa420e844a81390fb1158a9ec614f4bad55/node_modules/rxjs/internal/util/subscribeToResult.js:10:45)
    at MergeMapSubscriber._innerSub (/Users/ssieber/Library/Caches/Yarn/v3/npm-rxjs-6.3.3-3c6a7fa420e844a81390fb1158a9ec614f4bad55/node_modules/rxjs/internal/operators/mergeMap.js:82:29)
    at MergeMapSubscriber._tryNext (/Users/ssieber/Library/Caches/Yarn/v3/npm-rxjs-6.3.3-3c6a7fa420e844a81390fb1158a9ec614f4bad55/node_modules/rxjs/internal/operators/mergeMap.js:76:14)

Desired functionality

I'd like it work ;) - Specifically when I'm not using yarn pnp, I get this:

yarn run v1.12.0
warning ../package.json: No license field
$ ng build
                                                                                          
Date: 2018-10-03T18:44:42.889Z
Hash: df3a0ea990e4eeeb9455
Time: 8768ms
chunk {main} main.js, main.js.map (main) 25.6 kB [initial] [rendered]
chunk {polyfills} polyfills.js, polyfills.js.map (polyfills) 227 kB [initial] [rendered]
chunk {runtime} runtime.js, runtime.js.map (runtime) 5.22 kB [entry] [rendered]
chunk {styles} styles.js, styles.js.map (styles) 15.6 kB [initial] [rendered]
chunk {vendor} vendor.js, vendor.js.map (vendor) 3.54 MB [initial] [rendered]
✨  Done in 12.48s.

Mention any other details that might be useful

This is apparently caused the build tool implementing it's own dependency resolution. Yarn pnp provides an alternative dependency resolution that eschews node_modules. It's an experimental feature. I've filed an issue in their repo, but they directed me here, since you guys seem to be using custom resolution. See yarnpkg/yarn#6482

[arcanis]

As mentioned, Angular seems to do the resolution by itself, which causes issues when operating within exotic installation schemes where some assumptions don't necessarily hold. Fixes look quite easy though!

I've made a few quick fixes to see what's the minimal amount of changes needed to make it work, and listed them below. Let me know what you think, and if you want to setup a call to discuss all this in more details I'm all for it! Please ping me by email or Twitter or this thread 🙂

Missing dependencies

Plug'n'Play detected various unsafe behaviors that probably should be fixed whatever you choose to do:

• @angular/cli omits to list the tslib dependency in its package.json
• @angular/compiler-cli also omits tslib
• @angular-devkit/build-angular omits loader-utils and semver
• webpack-subresource-integrity relies on a hack (opened html-webpack-plugin should be a peer dependency, or not directly require()'d waysact/webpack-subresource-integrity#90)

@angular-devkit/build-angular

Cannot locate node_modules

Hmm, can I skip this one? That's kinda expected, I'd say 😛

Overall there's a bunch of node_modules assumptions there, so I guess it would make more sense for you to give it a look yourself - from what I gather it wouldn't be too much work to have something more generic. The cache folder can be set to .pnp/pkgdata/angular when you're in a PnP environment (instead of node_modules/something, which is risky), and the Webpack loaders can be easily (really!) loaded through the use of the pnp-webpack-plugin package. It's really just two extra lines in your configuration.

require-project-module.js

Basically just have to add a guard to directly use the PnP API when operating under a PnP environment (I can see the reticence to special case, but keep in mind this API is precisely meant to offer a common abstraction to how various projects could decide to layout the dependencies - in this sense it's more generic than resolve, which only supports node_modules).

function resolveProjectModule(root, moduleName) {
    if (process.versions.pnp) {
        return require('pnpapi').resolveRequest(moduleName, `${root}/`);
    } else {
        return resolve.sync(moduleName, { basedir: root });
    }
}

Note that this has to be done in @angular-devkit/build-angular as well (but I supposed the source code is mutualized between those two locations?).

@angular-devkit/core/node/resolve.js

Similarly to the previous note, a small edit is needed:

function resolve(x, options) {
    // ...
    const extensions = options.extensions || Object.keys(require.extensions);
    const basePath = options.basedir;
    if (process.versions.pnp) {
        const orig = x;
        const pnpapi = require('pnpapi');
        try {
            if (options.resolvePackageJson)
                x += `/package.json`;
            x = pnpapi.resolveRequest(x, `${basePath}/`, {extensions});
        } catch (error) {
            x = orig;
        }
    }
    // ...
}

[samsieber]

Small update - yarn plug 'n play (pnp) is now usable on the latest stable yarn (1.12.3). It isn't widely advertised yet, so it's still low profile / probably still experimental.

[whyboris]

Confirming error persists with the latest versions:

• yarn 1.13.0
• angular 7.1.4
• angular-cli 7.1.4
• @angular-devkit/build-angular 0.11.4

Could not find module "@angular-devkit/build-angular" from "/Users/byakubchik/Desktop/yarn/publish".
Error: Could not find module "@angular-devkit/build-angular" from "/Users/byakubchik/Desktop/yarn/publish".

[samsieber]

Another update - yarn pnp has official documentation now: https://yarnpkg.com/en/docs/pnp

Also, yarn is planning on making pnp the default mode in yarn 2.0: yarnpkg/yarn#6953

[cveld]

Any update on if yarn pnp support will get added to Angular. And if so, when?