TPI: debugging, multiroot, and large workspace testing with unittest

[eleanorjboyd]

Refs (overarching issue): #19098

• macOS @bpasero
• linux @karthiknadig
• windows @Yoyokrazy
• anyOS @paulacamargo25
  Complexity: 4

Author: @eleanorjboyd

Create Issue

---

Thank you for doing this testing! I apologize for how long it is and please let me know if you have any questions!
Prerequisites:

1. Use the following repo of tests or use your own repo of pytests https://github.com/eleanorjboyd/inc_dec_example_repo

Background Information:

This rewrite is for the underlying infrastructure of how python handles testing therefore all behavior visible to the user should be pretty much the same (minus maybe some error logs going in a better place or test names shortened). Therefore throughout this testing, you can reference the current testing in Python to compare if you think something doesn't look right (or you can always post here too!).

Steps:

Part 0: Setup

1. make sure you have the setting "python.experiments.optInto": ["All"], in your user settings json
2. set the log level to trace
3. go to the python output channel and search for pythonTestAdapter and something should come up that looks like this: 2023-05-26 11:12:04.297 [info] Experiment 'pythonTestAdapter' is active (this confirms you are actively in the experiment)
4. Now open the inc_dec_example_repo/inc_dec_example_test_suite  testing repo folder in vscode
5. use the create environment command from the command palette, select venv and whatever python version you want >=3.7
6. in the terminal run python -m pip install pytest
7. from the command palette run python: configure tests select unittest and "." (root directory) and test_*.py

Part 1: Debugging

1. Go to the testing explorer, and confirm all the tests are discovered and nested correctly by file structure.
2. add some break points to the unittest tests and check to make sure these break points are hit
   1. make sure you can do other normal debugging actions (stop, restart, step over, etc)

Part 2: Large repo

1. If you have a very large python repo definitely try it on that repo but if not you can use the provided repo inc_dec_example_repo/inc_dec_example_test_suite
   1. if you are using the provided repo: in the test_subtests.py file, change for i in range(0, 10): to be much larger like for i in range(0, 10000): and make sure it both works for discovery and run tests (the subtests should not be displayed in the test explorer after discovery only after run)
   2. if you are using your own: make sure all tests can be discovered and run as expected within your large repo when set to use unittests

Part 3: Multiroot Workspace

1. now you are going to open inc_dec_example_repo/folder3 but open it as a multiroot workspace (open the file testing3/testing3.code-workspace)
2. go to the testing panel, make sure that the tests are set up with the correct setting per workspace. This means that testFolder1 and testFolder3 are pytest and testFolder2 is unittest and they should all be displayed the correct tests based on this type.

[bpasero]

I ran into quite some issues (debugging not working, multi-root tests not showing, could not test large test suite), not sure if actually bugs or setup related. 👍 for the instructions, that got me setup quite fast.

[eleanorjboyd]

@bpasero did you successfully see the line in the log Experiment 'pythonTestAdapter' is active? Could you create an issue for the debugging and include the logs so I could try and deduce what might be wrong? That might give insight into the other steps not working so we could start there. Thanks

[bpasero]

@eleanorjboyd oh I reported all issues, I am not sure why they are not linked back to this issue 🤔

I swear I was seeing an entry for pythonTestAdapter in the output, now somehow I am not seeing that anymore 🤔

[bpasero]

Here are my issues: https://github.com/microsoft/vscode-python/issues/created_by/bpasero

They seem to reference not the right issue though.

[eleanorjboyd]

I swear I was seeing an entry for pythonTestAdapter in the output, now somehow I am not seeing that anymore 🤔

I am no longer seeing pythonTestAdapter in the output and @cwebster-99 is also having this experience. Currently, the best way for people to check right now is running a test (any test, unittest or pytest) and then checking for [debug] executionFactory defined in your logs.

[Yoyokrazy]

Just set things up, I am successfully seeing pythonTestAdapter in my python logs

edit with settings.json contents:

    // python settings
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true,
    "python.analysis.logLevel": "Trace",
    "python.experiments.optInto": [
        "All"
    ],
    // test suite args
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        ".",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestArgs": [
        "."
    ]
}

[Yoyokrazy]

Also hitting #21336