Refactor pytest discovery code in extension

[drorata]

Environment data

• VS Code version: 1.30.1 (dea8705087adb1b5e5ae1d9123278e178656186a)
• Extension version (available under the Extensions sidebar): 2018.12.1
• OS and version: macOS 10.14.2 (18C54)
• Python version (& distribution if applicable, e.g. Anaconda): 3.6.8 (Anaconda)
• Type of virtual environment used : conda
• Relevant/affected Python packages and their versions: pytest 4.1.0

Local setting is:

{
    "python.pythonPath": "/Users/dror/anaconda3/envs/tmp/bin/python",
    "python.unitTest.pyTestArgs": [
        "."
    ],
    "python.unitTest.unittestEnabled": false,
    "python.unitTest.nosetestsEnabled": false,
    "python.unitTest.pyTestEnabled": true
}

Expected behaviour

When running Python: Discover Unit Tests I expect not to receive an error message.

Actual behaviour

When I run pytest at the root of my project (from the terminal) everything works as expected: all tests are found and executed. However, when I try to run Python: Discover Unit Tests, I get the following error No tests discovered, please check the configuration settings for the tests.. The output is odd:

============================= test session starts ==============================
platform darwin -- Python 3.6.8, pytest-4.1.0, py-1.7.0, pluggy-0.8.0
rootdir: /Users/dror/tmp, inifile:
collected 2 items
<Module test_foo.py>
  <Function test_a>
  <Function test_b>

========================= no tests ran in 0.01 seconds =========================```

Moreover, when I do Python: Run All Tests I get the above output, followed by actually running the tests and passing them (as if I was running pytest directly in the console).

Steps to reproduce:

1. Create a conda virtual environment (tmp) and install pytest using pip
2. Create foo.py and test_foo.py (see below).
3. Run pytest in the console and see the two tests pass.
4. In vscode, run Python: Discover Unit Tests.
5. Then run Python: Run All Unit Tests.

#  foo.py
def bar(x):
    return x + 10

############################################

#  test_foo.py 
import foo


def test_a():
    assert 1 == 1


def test_b():
    assert foo.bar(5) == 15

Logs

Output for Python in the Output: see above.

The output of running the tests is:

============================= test session starts ==============================
platform darwin -- Python 3.6.8, pytest-4.1.0, py-1.7.0, pluggy-0.8.0
rootdir: /Users/dror/tmp, inifile:
collected 2 items
<Module test_foo.py>
  <Function test_a>
  <Function test_b>

========================= no tests ran in 0.01 seconds =========================
============================= test session starts ==============================
platform darwin -- Python 3.6.8, pytest-4.1.0, py-1.7.0, pluggy-0.8.0
rootdir: /Users/dror/tmp, inifile:
collected 2 items

test_foo.py ..                                                           [100%]

 generated xml file: /var/folders/w6/6m9g0bjn1q17bqdw33mqdqb40000gp/T/tmp-20946jC8DYWY6C8bR.xml 
=========================== 2 passed in 0.01 seconds ===========================

Output from Console under the Developer Tools panel: I couldn't find relevant information here.

[michaelmalonenz]

I can confirm I am also seeing the behaviour after the pytest 4.1.0 upgrade. It worked with pytest 4.0.1

[DonJayamanne]

Previous versions would contain quotes around file names:

<Module 'test_foo.py'>
  <Function 'test_a'>
  <Function 'test_b'>

New version of PyTest doesn't.
Hence the code used to parse the output fails.

[drorata]

I can confirm that downgrading pytest to 4.0.1 solves the problem. It seems, if I get it right, that the cause on pytest's side is:

https://github.com/pytest-dev/pytest/blob/a4c426b1a891a22ae1b63d0a0fa2dcdf690e69db/src/_pytest/terminal.py#L614-L617

[DonJayamanne]

The solution is fairly simple:

• Modify test discovery to use python code directly to discover tests
• Pass the pytest args to the python code.
• Use python code as follows:

import pytest
class MyPlugin(object):
    def pytest_collection_modifyitems(sessino, config, items):
        print(items, "items from collected list")
    def pytest_collection_finish(self, session):
        print(list(session.items), "items from session")

pytest.main(["-v", "tests", "--collect-only"], plugins=[MyPlugin()])

• Should be a simple enough change.

[drorata]

@DonJayamanne Is this a local solution or the approach to solve the issue on vscode-python's side?

[brettcannon]

@drorata we plan to solve it in the extension. Basically pytest does not guarantee stdout format stability so we have to change on our end.

[potens1]

I don't know if this can help in all cases, but using the -q flag of pytest output a more structured format (but I did not tried with tests classes, instances and so on, only functions):

$ pytest --collect-only -q
tests/test_vscode.py::test_error1
tests/test_vscode.py::test_error2
tests/test_vscode.py::test_error4

[DonJayamanne]

• [ ] Write python code to discover tests Pytest discovery in python code #4033
• [ ] CI to test python code Azdo CI to run python tests #4034
• [ ] Refactor pytest test discovery in extension Refactor pytest discovery code in extension to use python #4035

[kondratyev-nv]

I have fixed a similar issue in test explorer extension for Python. I have used the approach similar to the one used for unittest discovery https://github.com/DonJayamanne/pythonVSCode/blob/master/src/client/unittests/unittest/services/parserService.ts. To discover tests I used pytest_collection_finish hook, see https://github.com/kondratyev-nv/vscode-python-test-adapter/pull/48/files#diff-8fe02f614ffe36db100b8ad8758546b3R24.

Hope this might be of help!

[DonJayamanne]

No longer a P1 as we have a fix for existing users

[luabud]

@ericsnowcurrently can this issue be considered as done? if not, what's left here?

[ericsnowcurrently]

Yeah, the pytest adapter solves this.