No coverage using v11.0.0, ava, and nyc

[chriszarate]

After following the instructions in the release notes and nyc's updated README, I am unable to get v11.0.0 to generate coverage with nyc v6.6.1. Strangely, however, simply downgrading __coverage__ to v1.11.111 fixes the issue. I'm frankly not sure if this is an issue with __coverage__, nyc, or my own error.

Here is an extremely simple proof-of-problem that will hopefully allow anyone to replicate the issue:
https://github.com/chriszarate/nyc-coverage-11.0.0

The folder coverage-11.0.0 demonstrates the problem. The output of npm test shows an empty coverage report; the .nyc_output folder is created but is empty. To replicate:

cd coverage-11.0.0
npm i
npm test

The folder coverage-1.11.111 demonstrates working coverage output with the only change being a downgrade to v1.11.111. The .nyc_output contains the expected JSON output.

Any guidance you could provide on how to troubleshoot this myself would be appreciated. Please let me know if this should be filed with nyc or elsewhere.

[chriszarate]

$ node -v
v6.2.1
$ npm -v
3.9.6

[dtinth]

Thanks for a detailed report and a test case. Much appreciated! 👍
I’ve replicated your problem in my machine. Sorry that this version causes you a trouble.

It seems that ava does a chdir() into the test folder. Therefore, the source file is outside the current directory. Unfortunately, test-exclude has been hardcoded to not instrument any file outside current working directory. @bcoe, can you look into this problem please?

For now, you can stay with v1.11.111, because v11.0.0 did not contain any new feature.

[chriszarate]

Thanks for the quick diagnosis! I'll stick with v1.11.111 for now and look forward to continued progress in ES2015-coverage-land. Thanks for all your great work!

[bcoe]

@dtinth @chriszarate I wonder if this chdir() behavior has always been the case with ava; I believe it was the ava folks (@jamestalmage, @novemberborn) who helped build out the set of exclusion rules ... not instrumenting outside the current working folder solves potential problems with symlinked folders I believe.

Open to suggestions about a better approach, thoughts @jamestalmage?

[jamestalmage]

We are going to revert AVA's chdir behavior to always set it to the package.json directory. The current behavior has been around for a long time, but we regret it. Its caused numerous headaches.

[jamestalmage]

@bcoe. We should probably be analyzing test exclusion from the package.json dir, using pkg-up.

Unless they specify exclusion patterns on the command line. Then it should be from the directory the nyc CLI was first executed.

We should basically never be using process.cwd() except to set some initial global environment variable. Wrapped processes need to be able to change directories without breaking things.

[bcoe]

@jamestalmage if you have a potential implementation in mind, want to take a stab at a patch and submit it to test-exclude? also really curious to hear your thoughts on some of the istanbul threads currently taking place.

[jamestalmage]

also really curious to hear your thoughts on some of the istanbul threads currently taking place

Got links?

[bcoe]

@jamestalmage:

istanbuljs/nyc#282
istanbuljs-archived-repos/istanbul-lib-instrument#5

[bcoe]

@chriszarate this should be fixed in:

https://www.npmjs.com/package/babel-plugin-istanbul#integrations

give it a spin. babel-plugin-istanbul is a port of @dtinth's amazing work with __coverage__.

[chriszarate]

@bcoe I think my specific issue won't be fixed until ava updates its chdir behavior in the next release. Just to be sure I added a test case for babel-plugin-istanbul and ava and nyc still produces no output:

https://github.com/chriszarate/nyc-coverage-11.0.0/tree/master/istanbul-1.0.0

This is all very exciting work, though, and I'll be giving babel-plugin-istanbul a spin soon. Thanks!

I'm going to close this issue since we've established that my specific issue is caused by the chdir behavior in ava.