Ideas for speeding up and including flakey tests in PR

[rchiodo]

This should hopefully make tests run fast enough to run on a PR.

It does this:

• Runs each test file in a parallel
• Splits the functional tests into four groups and runs those in parallel on four separate jobs

Some preliminary timing

• Normal functional run on my box (without any splitting) - 15 minutes
• Functional run with each file running as separate process - 11 minutes

That was without Jupyter though. This PR should try the same thing with Jupyter

• Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR).
• Title summarizes what is changing.
• Has a news entry file (remember to thank yourself!).
• Appropriate comments and documentation strings in the code.
• Has sufficient logging.
• Has telemetry for enhancements.
• Unit tests & system/integration tests are added/updated.
• Test plan is updated as appropriate.
• package-lock.json has been regenerated by running npm install (if dependencies have changed).
• The wiki is updated with any design decisions/details.

[codecov-commenter]

Codecov Report

Merging #14265 into main will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main   #14265      +/-   ##
==========================================
- Coverage   59.96%   59.94%   -0.03%     
==========================================
  Files         706      706              
  Lines       39180    39180              
  Branches     5681     5681              
==========================================
- Hits        23494    23485       -9     
- Misses      14449    14455       +6     
- Partials     1237     1240       +3     

Impacted Files	Coverage Δ
...very/locators/services/virtualenvwrapperLocator.ts	76.92% <0.00%> (-15.39%)	⬇️
src/client/common/utils/platform.ts	60.00% <0.00%> (-12.00%)	⬇️
.../pythonEnvironments/common/externalDependencies.ts	57.57% <0.00%> (-3.04%)	⬇️
src/client/linters/pydocstyle.ts	86.66% <0.00%> (-2.23%)	⬇️
src/client/datascience/debugLocationTracker.ts	76.56% <0.00%> (-1.57%)	⬇️
src/client/common/process/proc.ts	14.49% <0.00%> (-0.73%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef44f29...72836e1. Read the comment docs.

[rchiodo]

I can't get Azdo to run again. Uggh.

[DonJayamanne]

Weed can optionally run this same workflow in ci (commits to main),
This way we have three running on main branch for every commit, and we don't need to copy paste this (just a thought, copy paste sux)

[rchiodo]

I believe that's a 'push' right? Added that.

[rchiodo]

@karthiknadig we're splitting our DS functional tests out into github actions. This is so we can make them run faster and run with real juptyer during a PR. RIght now they won't be required, but this is the first step in the split.

Thought you might want to know since they'll be running on core submissions too. Although I think I can prevent that with an inclusion list?

[rchiodo]

Woot! It worked. 23 minutes for the longest one now. That's the native editor tests.

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (main@ef44f29). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main   #14265   +/-   ##
=======================================
  Coverage        ?   59.94%           
=======================================
  Files           ?      706           
  Lines           ?    39180           
  Branches        ?     5681           
=======================================
  Hits            ?    23485           
  Misses          ?    14455           
  Partials        ?     1240           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef44f29...217de7b. Read the comment docs.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
0 Code Smells

No Coverage information
0.0% Duplication