coverage fails with no-data-collected when run from tox -p auto, but not when run serially #950

mgedmin · 2020-03-06T08:11:04Z

Describe the bug

I've this very weird situation:

git clone https://github.com/zopefoundation/zope.file && cd zope.file
tox -e coverage # works fine!
tox -p auto -e py38,coverage # coverage is shown as 0% and there's a

Coverage.py warning: No data was collected. (no-data-collected)

To Reproduce

How can we reproduce the problem? Please be specific.

git clone https://github.com/zopefoundation/zope.file
cd zope.file
tox -p auto -e coverage,py38

What version of Python are you using?

3.7.5, but I had the same problem with 2.7.17 (with basepython = python2.7 in the [testenv:coverage] section in tox.ini)

What version of coverage.py are you using? The output of coverage debug sys is helpful.

5.0.3

-- sys -------------------------------------------------------
                        version: 5.0.3
                       coverage: /home/mg/src/zopefoundation/zope.file/.tox/coverage/lib/python3.7/site-packages/coverage/__init__.py
                         tracer: -none-
                        CTracer: available
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: .coveragerc
                   configs_read: /home/mg/src/zopefoundation/zope.file/.coveragerc
                    config_file: /home/mg/src/zopefoundation/zope.file/.coveragerc
                config_contents: '[run]\nsource = src\n\n[report]\nexclude_lines =\n    pragma: no cover\n'
                      data_file: -none-
                         python: 3.7.5 (default, Nov 20 2019, 09:21:52) [GCC 9.2.1 20191008]
                       platform: Linux-5.3.0-40-generic-x86_64-with-Ubuntu-19.10-eoan
                 implementation: CPython
                     executable: /home/mg/src/zopefoundation/zope.file/.tox/coverage/bin/python
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 18570
                            cwd: /home/mg/src/zopefoundation/zope.file
                           path: /home/mg/src/zopefoundation/zope.file/.tox/coverage/bin
                                 /usr/lib/python37.zip
                                 /usr/lib/python3.7
                                 /usr/lib/python3.7/lib-dynload
                                 /home/mg/src/zopefoundation/zope.file/.tox/coverage/lib/python3.7/site-packages
                                 /home/mg/src/zopefoundation/zope.file/src
                    environment: PYTHONSTARTUP = /home/mg/.python
                   command_line: .tox/coverage/bin/coverage debug sys
                sqlite3_version: 2.6.0
         sqlite3_sqlite_version: 3.29.0
        sqlite3_compile_options: COMPILER=gcc-9.2.1 20191008
                                 ENABLE_COLUMN_METADATA
                                 ENABLE_DBSTAT_VTAB
                                 ENABLE_FTS3
                                 ENABLE_FTS3_PARENTHESIS
                                 ENABLE_FTS3_TOKENIZER
                                 ENABLE_FTS4
                                 ENABLE_FTS5
                                 ENABLE_JSON1
                                 ENABLE_LOAD_EXTENSION
                                 ENABLE_PREUPDATE_HOOK
                                 ENABLE_RTREE
                                 ENABLE_SESSION
                                 ENABLE_STMTVTAB
                                 ENABLE_UNLOCK_NOTIFY
                                 ENABLE_UPDATE_DELETE_LIMIT
                                 HAVE_ISNAN
                                 LIKE_DOESNT_MATCH_BLOBS
                                 MAX_SCHEMA_RETRY=25
                                 MAX_VARIABLE_NUMBER=250000
                                 OMIT_LOOKASIDE
                                 SECURE_DELETE
                                 SOUNDEX
                                 TEMP_STORE=1
                                 THREADSAFE=1
                                 USE_URI

What versions of what packages do you have installed? The output of pip freeze is helpful.

beautifulsoup4==4.8.2
BTrees==4.6.1
cffi==1.14.0
coverage==5.0.3
persistent==4.6.0
pycparser==2.20
python-gettext==4.0
pytz==2019.3
six==1.14.0
soupsieve==2.0
transaction==3.0.0
waitress==1.4.3
WebOb==1.8.6
WebTest==2.0.34
WSGIProxy2==0.4.6
zc.lockfile==2.0
ZConfig==3.5.0
zdaemon==4.3
ZODB==5.5.1
zodbpickle==2.0.0
zope.annotation==4.7.0
zope.app.appsetup==4.1.0
zope.app.basicskin==4.0.0
zope.app.http==4.0.1
zope.app.pagetemplate==4.0.0
zope.app.principalannotation==4.0.0
zope.app.publication==4.3.2
zope.app.wsgi==4.1.0
zope.applicationcontrol==4.2.0
zope.authentication==4.4.0
zope.browser==2.3
zope.browsermenu==4.4
zope.browserpage==4.4.0
zope.browserresource==4.4
zope.cachedescriptors==4.3.1
zope.component==4.6
zope.configuration==4.3.1
zope.container==4.3.0
zope.contenttype==4.5.0
zope.copy==4.2
zope.copypastemove==4.1.0
zope.datetime==4.2.0
zope.deferredimport==4.3.1
zope.deprecation==4.4.0
zope.dottedname==4.3
zope.dublincore==4.2.0
zope.error==4.5.0
zope.event==4.4
zope.exceptions==4.3
-e [email protected]:zopefoundation/zope.file.git@2833ea32239f5c0d9e2f8a4f51512a2c67b32457#egg=zope.file
zope.filerepresentation==4.2.0
zope.formlib==4.7.0
zope.hookable==5.0.0
zope.i18n==4.7.0
zope.i18nmessageid==5.0.0
zope.interface==4.7.1
zope.lifecycleevent==4.3
zope.location==4.2
zope.login==2.1.0
zope.mimetype==2.4.0
zope.minmax==2.2.0
zope.pagetemplate==4.5.0
zope.password==4.3.1
zope.principalannotation==4.3.0
zope.principalregistry==4.2.0
zope.processlifetime==2.3.0
zope.proxy==4.3.3
zope.publisher==5.1.1
zope.schema==4.9.3
zope.security==5.1.0
zope.securitypolicy==4.3.1
zope.session==4.3.0
zope.site==4.2.2
zope.size==4.3
zope.tal==4.4
zope.tales==5.0.1
zope.testbrowser==5.5.1
zope.testing==4.7
zope.testrunner==5.1
zope.traversing==4.3.1

What code are you running? Give us a specific commit of a specific repo that we can check out.

https://github.com/zopefoundation/zope.file/tree/2833ea32239f5c0d9e2f8a4f51512a2c67b32457

What commands did you run?

tox -p auto, but limiting the test environments to just two is sufficient to reproduce this, e.g. tox -e py27,coverage -p auto or tox -e py38,coverage -p auto

Expected behavior

96% coverage, which is what I get when I run tox -e coverage

Additional context

There are subprocesses involved in the test script, due to zope.testrunner's feature that can run some tests in a subprocess. I'm not entirely sure how coverage tracking works with this usually.

I've tried to see if this might be a race condition due to additional system load from other tests running in parallel, but I couldn't reproduce this by running tox -e py38 >&/dev/null & tox -e coverage.

The text was updated successfully, but these errors were encountered:

mgedmin · 2020-03-06T08:23:58Z

There are subprocesses involved in the test script, due to zope.testrunner's feature that can run some tests in a subprocess. I'm not entirely sure how coverage tracking works with this usually.

This is a red herring: I've got rid of them in zopefoundation/zope.file@1e88bb9 and yet this problem persists.

cjw296 · 2020-03-06T08:39:27Z

Certainly smells like subprocesses, have you pored through https://coverage.readthedocs.io/en/coverage-5.0.3/subprocess.html in great detail?

Ones that often catch me:

the magic .pth file either not being present or not being picked up correctly
something stripping COVERAGE_PROCESS_START out of the environment which isn't recording coverage
something killing the subprocess before the atexit handler fires, so coverage data isn't saved to disk.

mgedmin · 2020-03-06T08:43:09Z

I've reproduced this without subprocesses (or threads).

I'm now staring at an strace log and trying to understand anything. I see a bunch of openat(AT_FDCWD, ".coverage", O_RDWR|O_CREAT|O_CLOEXEC, 0644) = 3 from the same pid (zope-testrunner), with reads and writes and lock and whatnots, and then there's a sudden

write(2, "Coverage.py warning: No data was collected. (no-data-collected)\n", 64)

mgedmin · 2020-03-06T08:46:07Z

Doing a .dump on the .coverage fille shows that it knows what source files there were, but it has recorded no line_bits at all.

mgedmin · 2020-03-06T09:09:28Z

... maybe I should strace the working process and compare?

Uh, the working process opens .coverage five times. The broken one does that 19 times.

Wow the line_bits representation is amazingly compact!

I don't think I'll be able to extract anything useful from strace.

COVERAGE_DEBUG=tracing sheds more light on this:

with tox -e py38,coverage -p auto I get a bunch of Not tracing '<doctest adapters.rst[16]>': falls outside the --source trees
with tox -e coverage -p auto I get a bunch of Tracing '<doctest adapters.rst[16]>' messages

so that's interesting!

mgedmin · 2020-03-06T09:14:10Z

coveragepy is innocent and has been unfairly framed by a bug in tox!

When I run tox -e coverage -p auto, and then I look at .tox/coverage/bin/pip list, I see that tox's [testenv:coverage] usedevelop = true was implemented and zope.file is installed in editable mode.

When I run tox -e py38,coverage -p auto and then I look at .tox/coverage/bin/pip list, I see that tox ignored the usedevelop setting and did a plain pip install, so coverage naturally doesn't see anything being executed from inside its specified source path ([run] source = src in .coveragerc)!

sysid · 2022-08-06T09:52:31Z

@mgedmin I have got the same issue. Did you find a solution?
Would be much appreciated, thank you!

sysid · 2022-08-06T09:58:29Z

Found the solution, for future reference in case someone is ending up here:
usedevelop = true was missing.

mgedmin added the bug Something isn't working label Mar 6, 2020

mgedmin closed this as completed Mar 6, 2020

nedbat added the not our bug The problem was elsewhere label Mar 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

coverage fails with no-data-collected when run from tox -p auto, but not when run serially #950

coverage fails with no-data-collected when run from tox -p auto, but not when run serially #950

mgedmin commented Mar 6, 2020 •

edited

Loading

mgedmin commented Mar 6, 2020

Uh oh!

cjw296 commented Mar 6, 2020

Uh oh!

mgedmin commented Mar 6, 2020

Uh oh!

mgedmin commented Mar 6, 2020

Uh oh!

mgedmin commented Mar 6, 2020

Uh oh!

mgedmin commented Mar 6, 2020 •

edited

Loading

Uh oh!

sysid commented Aug 6, 2022

Uh oh!

sysid commented Aug 6, 2022

Uh oh!

Uh oh!

coverage fails with no-data-collected when run from tox -p auto, but not when run serially #950

coverage fails with no-data-collected when run from tox -p auto, but not when run serially #950

Comments

mgedmin commented Mar 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mgedmin commented Mar 6, 2020

Uh oh!

cjw296 commented Mar 6, 2020

Uh oh!

mgedmin commented Mar 6, 2020

Uh oh!

mgedmin commented Mar 6, 2020

Uh oh!

mgedmin commented Mar 6, 2020

Uh oh!

mgedmin commented Mar 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sysid commented Aug 6, 2022

Uh oh!

sysid commented Aug 6, 2022

Uh oh!

mgedmin commented Mar 6, 2020 •

edited

Loading

mgedmin commented Mar 6, 2020 •

edited

Loading