python.testing.pytestArgs: passing config file path when it's in a subfolder consumes 100% of a CPU

[pawamoy]

Discussed in #23386

^{Originally posted by pawamoy May 8, 2024}
VSCode keeps spawning a pytest process to collect tests (apparently).

    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--config-file=config/pytest.ini"
    ],

It runs it in my current .venv: /absolute/path/to/.venv/bin/python -m pytest -p vscode_pytest --collect-only --config-file=config/pytest.ini.

The process never stops and consumes one CPU entirely.

My pytest config:

[pytest]
python_files =
  test_*.py
  *_test.py
  tests.py
addopts =
  --cov
  --cov-config config/coverage.ini
testpaths =
  tests

# action:message_regex:warning_class:module_regex:line
filterwarnings =
  error
  # TODO: remove once pytest-xdist 4 is released
  ignore:.*rsyncdir:DeprecationWarning:xdist
  ignore:.*slated for removal in Python:DeprecationWarning:.*

If I run the command myself:

% .venv/bin/python -m pytest -p vscode_pytest --collect-only --config-file=config/pytest.ini
Traceback (most recent call last):
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 865, in import_plugin
    __import__(importspec)
ModuleNotFoundError: No module named 'vscode_pytest'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/pytest/__main__.py", line 7, in <module>
    raise SystemExit(pytest.console_main())
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 206, in console_main
    code = main()
           ^^^^^^
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 159, in main
    config = _prepareconfig(args, plugins)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 346, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/helpconfig.py", line 106, in pytest_cmdline_parse
    config = yield
             ^^^^^
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1152, in pytest_cmdline_parse
    self.parse(args)
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1501, in parse
    self._preparse(args, addopts=addopts)
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1384, in _preparse
    self.pluginmanager.consider_preparse(args, exclude_only=False)
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 794, in consider_preparse
    self.consider_pluginarg(parg)
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 817, in consider_pluginarg
    self.import_plugin(arg, consider_entry_points=True)
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 867, in import_plugin
    raise ImportError(
  File "/home/pawamoy/data/dev/griffe/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 865, in import_plugin
    __import__(importspec)
ImportError: Error importing plugin "vscode_pytest": No module named 'vscode_pytest'

Without the vscode_pytest plugin:

% .venv/bin/python -m pytest --collect-only --config-file=config/pytest.ini 
========================================================================================= test session starts =========================================================================================
platform linux -- Python 3.12.1, pytest-8.2.0, pluggy-1.5.0
Using --randomly-seed=153736517
rootdir: /home/pawamoy/data/dev/griffe/config
configfile: pytest.ini
plugins: cov-5.0.0, randomly-3.15.0, xdist-3.6.1
collected 777 items
... (truncated)
===== 777 tests collected in 2.97s =====

Any idea why the process hangs and consumes an entire CPU?

Could be related to microsoft/pylance-release#4218 🤷

---

As mentioned in the discussion, I isolated the issue. It only happens when my pytest config file is in a subdirectory.

This works fine:

    "python.testing.pytestArgs": [
        "--config-file=pytest.ini"
    ],

While this consumes a CPU and never finishes:

    "python.testing.pytestArgs": [
        "--config-file=config/pytest.ini"
    ],

[eleanorjboyd]

Hello! Thanks for this detailed bug report, I will investigate. Could you include your logs from the python output channel? For your logs, can you first set your log level to trace via theDeveloper: set log level command in the command palette then rerun the tests. I am curious what point in the process it gets stuck at and if the logs.

Also are you running the command yourself from the root of the repo right?

Finally are you able to see the same behavior for an arbitrary test- ie if you just write a pytest test which always passes and run with the config arg do you get the same behavior?

Sorry for all the questions! Thanks

[pawamoy]

Thanks a lot for your help!

Could you include your logs from the python output channel? For your logs, can you first set your log level to trace via theDeveloper: set log level command in the command palette then rerun the tests.

Sure. Note that I don't even run the tests, I only try to "collect" them (refreshing them in the Testing tab).

Trace logs:

2024-05-13 19:36:05.969 [debug] Testing: Manually triggered test refresh
2024-05-13 19:36:05.969 [debug] Testing: Clearing all discovered tests
2024-05-13 19:36:05.970 [debug] Testing: Forcing test data refresh
2024-05-13 19:36:05.970 [debug] Testing: Refreshing all test data
2024-05-13 19:36:05.971 [debug] Found cached env for /tmp/repro23411/.venv/bin/python
2024-05-13 19:36:05.975 [info] Discover tests for workspace name: repro23411 - uri: /tmp/repro23411
2024-05-13 19:36:05.975 [info] Running discovery for pytest using the new test adapter.
2024-05-13 19:36:05.976 [info] All environment variables set for pytest discovery for workspace /tmp/repro23411: REDACTED
2024-05-13 19:36:05.978 [debug] Found cached env for /tmp/repro23411/.venv/bin/python
2024-05-13 19:36:06.021 [debug] Running pytest discovery with command: -m pytest -p vscode_pytest --collect-only --config-file=config/pytest.ini for workspace /tmp/repro23411.
2024-05-13 19:36:06.025 [info] > ./.venv/bin/python -m pytest -p vscode_pytest --collect-only --config-file=config/pytest.ini
2024-05-13 19:36:06.025 [info] cwd: .
2024-05-13 19:36:06.343 [info] ============================= test session starts ==============================

2024-05-13 19:36:06.343 [info] platform linux -- Python 3.12.1, pytest-8.2.0, pluggy-1.5.0

2024-05-13 19:36:06.343 [info] rootdir: /tmp/repro23411/config
configfile: pytest.ini

2024-05-13 19:36:06.351 [info] collected 1 item

2024-05-13 19:36:06.352 [info] 
<Dir repro23411>
  <Dir tests>
    <Module test_hello.py>
      <Function test_hello>

2024-05-13 19:36:16.730 [debug] Found cached env for /tmp/repro23411/.venv/bin/python
2024-05-13 19:36:21.469 [error] Subprocess exited unsuccessfully with exit code null and signal SIGTERM on workspace /tmp/repro23411.
2024-05-13 19:36:21.470 [error] Subprocess exited unsuccessfully with exit code null and signal SIGTERM on workspace /tmp/repro23411. Creating and sending error discovery payload
2024-05-13 19:36:21.470 [error] pytest test discovery error for workspace:  /tmp/repro23411 
  
 The python test process was terminated before it could exit on its own, the process errored with: Code: null, Signal: SIGTERM for workspace /tmp/repro23411
2024-05-13 19:36:21.470 [info] ResultResolver EOT received for discovery.
2024-05-13 19:36:21.470 [debug] deferredTill EOT resolved for /tmp/repro23411
2024-05-13 19:36:21.470 [info] Disposing data receiver for /tmp/repro23411 and deleting UUID; pytest discovery.
2024-05-13 19:36:23.232 [debug] Found cached env for /tmp/repro23411/.venv/bin/python

The logs hang at <Function test_hello>, then I interrupt the pytest process and the logs continue with Subprocess exited unsuccessfully etc.

Also are you running the command yourself from the root of the repo right?

No, I'm not running anything myself: it's vscode-python which runs the pytest process to collect the tests. I can trigger it by clicking on the "Refresh tests" button in the "Testing" tab.

Finally are you able to see the same behavior for an arbitrary test- ie if you just write a pytest test which always passes and run with the config arg do you get the same behavior?

Yep, I reproduced this in a clean, minimal project:

% tree -a -L 2
.
├── config
│   ├── .pytest_cache
│   └── pytest.ini
├── tests
│   ├── __pycache__
│   └── test_hello.py
├── .venv
│   ├── bin
│   ├── CACHEDIR.TAG
│   ├── .gitignore
│   ├── lib
│   ├── .lock
│   └── pyvenv.cfg
└── .vscode
    └── settings.json

9 directories, 7 files

% cat .vscode/settings.json 
{
    "files.watcherExclude": {
        "**/.venv*/**": true,
        "**/venv*/**": true
    },
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--config-file=config/pytest.ini"
    ]
}

% cat config/pytest.ini 
[pytest]
python_files =
  test_*.py
testpaths =
  tests

% cat tests/test_hello.py 
def test_hello():
    assert True

Running the pytest command myself works fine:

% ./.venv/bin/python -m pytest --collect-only --config-file=config/pytest.ini 
==== test session starts ====
platform linux -- Python 3.12.1, pytest-8.2.0, pluggy-1.5.0
rootdir: /tmp/repro23411/config
configfile: pytest.ini
collected 1 item                                                                                                                                                                                              

<Dir repro23411>
  <Dir tests>
    <Module test_hello.py>
      <Function test_hello>

==== 1 test collected in 0.01s ====

[eleanorjboyd]

Hello! I have found the issue here. One thing I do to build the test tree seen in the test explorer is create nodes between session node (root) and test nodes. In this instance you can see in your logs that the root is within the config folder rootdir: /tmp/repro23411/config while the test files are inside the tests folder. Therefore the root dir is not a parent of the tests which was an assumption the plugin made. I will update it to check if the rootdir is not a parent and if so find the closest common path and making that the root (therefore making the tree look like below). Im putting together a fix but need a bit longer to create a test for this scenario.

[pawamoy]

That's awesome, thank you so much! Take all the time you need, no worries 😄 And thanks for the explanation, that makes total sense 🙂