Allow configuring DDC module type

[pulyaevskiy]

Looking at the source code I see that build_web_compilers has hardcoded amd module type when building for web.

Is there a way to override this default behavior?

I'm working on some packages which are designed for NodeJS and this requires common module type.

[pulyaevskiy]

Hm... I just realized that I probably need to write something like build_node_compiler similar to the web-specific package but with different configuration. I'm gonna try this...

[jakemac53]

The main work is changing the bootstrapping logic, which is fairly specific to require.js right now. Would require a bit of reorganization in this package to support it well but it wouldn't be too difficult. Making a node specific package is probably reasonable as well, we could extract the summary and module specific logic out of the build_web_compilers package as well which would help.

[pulyaevskiy]

Thanks @jakemac53 ! I already forked this repo and basing my work on it (it turns out there is a lot of logic around Modules in here).

If you don't mind I'd like keep this issue open to ask questions along the way?

[jakemac53]

Yep, sounds good

[jakemac53]

Also, we can chat on gitter here https://gitter.im/dart-lang/source_gen. We have a lot of follow-up meetings today but will be more available next week.

[pulyaevskiy]

Awesome, no rush at all. Here is a couple of questions I have so far.

For DDC modules to work with Node I'm thinking (after reading a lot) I need to put everything generated by DDC in a node_modules/packages somewhere. The question is are there any conventions around this?

I see build_runner creates .dart_tool/build/generated/{package} folders. Should I just move everything in .dart_tool/build/generated/* to build/node_modules/packages? This is probably not gonna work with incremental builds right?

Is there a way to customize output folders somehow?

Second question: I have a my_package/web/main.dart, which is also inside generated folder.
Ideally I'd like folder structure to look like this:

.dart_tool/
  - generated/
    - node_modules/
      - packages/
        - my_package/
    - web/
      - main.dart.js

Is it possible? Any advise would help a lot!

(also, I wanted to rename web/main.dart to node/main.dart but build_runner didn't pick up this folder for some reason).

[jakemac53]

For DDC modules to work with Node I'm thinking (after reading a lot) I need to put everything generated by DDC in a node_modules/packages somewhere. The question is are there any conventions around this?

This is a very interesting question - I think creating directories under node_modules conceptually maps to a packages folder, but I don't know if it is really intended to be mucked with manually. If you are using npm I don't know what the consequences might be.

I see build_runner creates .dart_tool/build/generated/{package} folders. Should I just move everything in .dart_tool/build/generated/* to build/node_modules/packages?

There is the --output <dir> flag which will create a merged output directory with a packages folder and whatever top level folders you have in your application (web, test, etc).

This should also essentially answer your 2nd question, about moving the web directory to the top level (I think the output dir matches what you want there).

(also, I wanted to rename web/main.dart to node/main.dart but build_runner didn't pick up this folder for some reason)

We have a hardcoded whitelist of common top level directories that are treated as inputs by default. To include other directories you need to create a build.yaml which looks something like this:

targets:
  $default:
    sources:
      - "lib/**"
      - "node/**"

[jakemac53]

It also looks like node supports a package.json file, as well as require.paths global which might be useful to avoid creating an actual node_modules directory.

[pulyaevskiy]

as well as require.paths global

Looks like it was removed from the API a while ago. Though it'd be great if it was there.
I will most likely have to override built-in require function just because DDC generates require calls to a some sort of idiomatic package name which doesn't match with actual file name of the module.

Something like this:

  // somewhere in main.dart.bootstrap.js
  var Module = require('module');
  var originalRequire = Module.prototype.require;

  Module.prototype.require = function () {
    var id = arguments['0'];
    if (id.startsWith('packages/') {
      // assume this is a DDC module, pseudo-code
      var path = modulePaths[id];
      arguments['0'] = path;
      return originalRequire.apply(this, arguments);
    }
    return originalRequire.apply(this, arguments);
  };

but I don't know if it is really intended to be mucked with manually. If you are using npm I don't know what the consequences might be.

Ideally I'd need to use npm to install JS packages next to the generated ones.
With --output=build I get a much better folder structure. Would it be considered ok to have a builder which generates build/packages.json (for npm package dependencies) and runs npm install so that those modules are installed in build/node_modules?

We have a hardcoded whitelist of common top level directories that are treated as inputs by default.

It didn't seem to work for me from the first try.
I have a fork of build_web_compilers, so I modified build.yaml in there as you suggested, might still be doing something wrong.

Thanks a lot for your help! I already got some positive results executing a basic DDC-compiled app in Node with require calls working.

[natebosch]

If you want to modify the files that you can load from your package you'll need to add a build.yaml in that package. Changing the build.yaml in build_web_compilers won't have an impact on the whitelist.

[jakemac53]

Would it be considered ok to have a builder which generates build/packages.json (for npm package dependencies) and runs npm install so that those modules are installed in build/node_modules?

I think generating a packages.json file is reasonable, but I would run npm install as a separate post-build step.

[matanlurey]

What is the actionable item for build? @jakemac53 @natebosch

[natebosch]

Two possibilities for what we could do in build:

• Pull out the module stuff from build_web_compilers into a separate package that we can reuse for this node stuff.
• Make the node module patterns an option to the compilers here.

If we can see a prototype of this running that would help us make a decision.

[matanlurey]

Sounds good. I'll label this as BWC for now.