Total rebuild test time in medium and large projects

[mattem]

Versions

    _                      _                 ____ _     ___
   / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
  / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
 / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
/_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
               |___/
@angular/cli: 1.4.7
node: 8.8.0
os: darwin x64
@angular/animations: 4.4.5
@angular/cdk: 2.0.0-beta.12
@angular/common: 4.4.5
@angular/compiler: 4.4.5
@angular/core: 4.4.5
@angular/flex-layout: 2.0.0-beta.9
@angular/forms: 4.4.5
@angular/http: 4.4.5
@angular/material: 2.0.0-beta.12
@angular/platform-browser: 4.4.5
@angular/platform-browser-dynamic: 4.4.5
@angular/router: 4.4.5
@angular/cli: 1.4.7
@angular/compiler-cli: 4.4.5
typescript: 2.3.4

Node v8.8.0
NPM v5.4.2
Mac OS High Sierra

Observed behavior

Running tests on medium and large projects take a long time to start, recompile and rerun tests.
We have are continually increasing the timeouts within the karma.conf file for the amount of time given to Chrome before it disconnects from no activity (increased both captureTimeout and browserNoActivityTimeout).

We have also experienced issues where, due to the timeout, we have multiple (headless) Chrome instances running the tests where Karma has started another instance (although this sounds like an issue with Karma, but I thought I'd mention it)

On our project in particular, the vendor bundle is ~15mb, which is taking a long time to load when running the test in debug mode (watching the network activity in Chrome before the tests start).

Opening this issue for a broader discussion with @filipesilva

[lucasvanhalst]

Upgrading to cli 1.5 / angular 5 should speed things up quite a lot. Alternatively (since this is a large project), disable sourcemaps (--sourcemaps=false), that should decrease the vendor bundle by a large amount and make the build faster.

[filipesilva]

We did some work to speed up rebuilds in 1.5.0 as @lucasvanhalst mentioned, but I know this is still a problem for many projects.

There's a couple of parts to this issue:

• rebuild time: the time it takes for the CLI to rebuild bundles when a change is made.
• karma test run time: the time it takes for karma to finish tests after a refreshing.
• JIT compilation time: the time it takes for the JIT compiler to compile component templates at runtime.

Let's look at them individually.

Rebuild time

This is the domain of the CLI build system. We have a highly customized karma webpack plugin that allows for a vendor chunk (#6160).

This helps in rebuilds by allowing your test chunk to be rebuilt separately from the vendor chunk. Generally speaking, smaller chunks are faster to rebuild so it helps to split.

You can actually further split your test chunks as well using System.import() (or just import() if you use "module": "esnext" in your tsconfig microsoft/TypeScript#16675 (comment)).

Modify your test files like this:

// end of ./src/test.ts

// // Then we find all the tests.
// const context = require.context('./', true, /\.spec\.ts$/);
// // And load the modules.
// context.keys().map(context);
// // Finally, start Karma to run the tests.
// __karma__.start();

// Import lazy test chunks.
declare var System: any;
Promise.all([
  System.import('./test-lazy-chunk-1.ts'),
  // More lazy chunk imports here.
]).then(() => {
  // Finally, start Karma to run the tests.
  __karma__.start();
});

// new file ./src/test-lazy-chunk-1.ts

// Then we find all the tests.
// Change the first argument of `.context()` to specify a subfolder like `./app/some-folder/`.
const context = (require as any).context('./', true, /\.spec\.ts$/);
// And load the modules.
context.keys().map(context);

// update ./src/tsconfig.spec.ts to include the lazy chunk

  "include": [
    "test-lazy-chunk-1.ts",
    "**/*.spec.ts",
    "**/*.d.ts"
  ]

Splitting your tests into chunk should make rebuilds faster.

Turning off sourcemaps also helps (by making rebundling faster) but then debugging will be hard.

Karma test run time

After a rebuild Karma will load your bundles and re-run all tests. So the more tests you have, the longer this will take. I think this is the problem most people actually have.

Jest solves this problem by only re-running stuff that needs to be run again. Last I checked we couldn't really plug the webpack build setup (just try to convert) into it so it's not an option.

The best you can do to limit this is limit the re-run tests via fdescribe of fit in your tests. Maybe we could run some advanced code transforms and do this for you? Would have to look into it.

JIT compilation time

After your code is rebuilt, it is loaded by the browser and tests are ran. Unit tests are always ran in JIT mode, which means that component templates are compiled in the browser.

This takes time. It's fairly imperceptible in small projects but becomes noticeable as template complexity and number increase.

Outside of unit tests, this is the time after the rebuild is done and your browser has reloaded, but your screen is white and your application hasn't actually loaded. That's the JIT compiler processing templates.

The solution to reducing this time is to use AOT in unit tests, which means that the build step would also compile templates. This is still experimental so we haven't added support to it in the CLI (see #8007 (comment)).

So for now there isn't much that you can do to reduce this time (apart from limiting the specs as listed in the Karma test run time section above).

[mattem]

Does 1.5.x require Angular v5? If so, I will have to schedule for our project to be updated (which is quite a large task in itself).

Rebuild time

Can you elaborate on the splitting the tests into chunks? Would these files be per test? (for example, foo.component.spec.ts) or a barrel containing imports for other test files (akin to a test suite in Junit I guess). Turning off sourcemaps for development and debugging in tests isn't really a viable option I'm afraid and we already go this route on CI.

Karma test run time

Understand, I'm not too worried about the test run time, we have approx. ~1500 tests, and I'm actually surprised how quickly this phase runs all things considered. During development, we will focus a suite or spec with fdescribe or fit.

JIT compilation time

Does the compiler compile all templates here, or just those run as part of the tests? They are declared in the beforeEach blocks, so I'm expecting it to only compile what is needed.
AoT compile for tests does sound promising.

Is there a way to minify just the vendor bundle for tests? If it's all 3rd party modules, I'm likely not going to debug it (that often hopefully!).

[filipesilva]

1.5 itself doesn't require Angular 5 but the new pipeline does. So effectively the rebuild improvements are only if you update both.

I'd try to just split them by folders really, to have a feel for it. So you could do something like:

const context = (require as any).context('./app/module-1/', true, /\.spec\.ts$/);

And so forth for the other top level modules.

Only templates for components being used at runtime are compiled.

No way to minify just the vendor bundle but I don't see how that would help. Minifying takes time, and if it's unchanged between builds it's also cached.

[mattem]

As an update...

We are in the process of moving our code and all its libraries into a mono-repo, as part of that process I've updated to Angular 5.1.0 and CLI 1.6.1. We are now running ~ 3000 specs. The code splitting using the lazy chunks have definitely helped the build / rebuild cycle, and loading a 'suite' of tests into the browser is much faster than loading the one bundle.

However, we am still hitting issues with the time taken to actually run the tests. A full run takes about 20 mins when it completes.
The main culprit seems to be our Form based tests. (a bit of background: we have user definable forms on the server side, any component could appear in a generated form, meaning the whole form has to be dynamic and injected at runtime).
Compiling the module that contains all these form components (100+) takes a long time, and as this is done per test... you can see where this is going. This seems okay when running in isolation (fit), but when running with the rest of the tests, it seems to slow down dramatically over time, even on seemingly unrelated tests, eventually causing Karma to disconnect and crash - on first glance it seems like a memory leak somewhere, but I have not done any investigation yet.

This is generally what we get on CI, with random timeouts when running locally (note that I've disabled a lot of tests still, hence the lower numbers)

HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 2017 of 2123 (skipped 27) SUCCESS (0 secs / 6 mins 50.281 secs)
20 12 2017 15:12:03.779:WARN [HeadlessChrome 0.0.0 (Linux 0.0.0)]: Disconnected (1 times)
HeadlessChrome 0.0.0 (Linux 0.0.0) ERROR
  Disconnectedundefined
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 2017 of 2123 (skipped 27) DISCONNECTED (8 mins 17.695 secs / 6 mins 50.281 secs)
HeadlessChrome 0.0.0 (Linux 0.0.0) ERROR
  Disconnectedundefined
HeadlessChrome 0.0.0 (Linux 0.0.0) ERROR
  Disconnectedundefined
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 2017 of 2123 (skipped 27) DISCONNECTED (8 mins 17.695 secs / 6 mins 50.281 secs)
�SUMMARY
  0: HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 2017 of 2123 (skipped 27) DISCONNECTED (8 mins 17.695 secs / 6 mins 50.281 secs)
test cases successful in all browsers

I've been looking into the suggestions posted in this issue: angular/angular#12409

I've also looked into running the tests in Jest and also following the progress on Bazel very closely here: https://github.com/alexeagle/angular-bazel-example

[filipesilva]

I remember @wardbell had some woes in running unit tests that were individually heavy, he might have some advice here.

The disconnects are bad though. That might need an increased timeout in Karma? Maybe that's not it though.

[mattem]

Apologies for the slow update here, been away for the holiday period.

After running though some investigation, I've managed to get our tests running without issue.
The main culprits for us were:

• Multiple tests with the same description (in the it block). When multiple tests exist with the same name, Karma sees only one of them as completed, although will run both, so we ended up situations where the Karma will report the following:

 0: HeadlessChrome 0.0.0 (Mac OS X 10.13.1): Executed 636 of 636 SUCCESS (0.851 secs / 0.36 secs)
  635 test cases successful in all browsers

A timeout occurs and then disconnects.

• For modules that take a long time to compile (like our Forms modules as described above), use the following wrapper inside a beforeAll to allow only compiling once.

import { TestBed, TestModuleMetadata } from '@angular/core/testing';

let _resetTestingModule;
export function configureTestingModule(moduleDef: TestModuleMetadata): Promise<typeof TestBed> {
  TestBed.resetTestingModule();
  const bed = TestBed.configureTestingModule(moduleDef);

  return bed.compileComponents().then(() => {
    _resetTestingModule = TestBed.resetTestingModule;
    TestBed.resetTestingModule = () => TestBed;

    return bed;
  });
}

export function resetTestingModule(): void {
  if (_resetTestingModule) {
    TestBed.resetTestingModule = _resetTestingModule;
  }

  TestBed.resetTestingModule();
  _resetTestingModule = undefined;
}

After these changes, our tests run in approx 10mins, including on CI.
I'm still looking at ways to improve this, such as using AoT summaries and Bazel, but haven't had much time to investigate there.

[filipesilva]

Heya, I'm closing this issue as the overall discussion seems to have run its course.