Smoke tests for assorted plugins #7721

ssbarnea · 2020-09-05T15:01:02Z

This aims to create a tox environment that tests that installation of
popular plugins does not break pytest.

This adds a new tox environment that tests that installation of popular plugins does not break pytest. We can add more as time passes.

Regression introduced in pytest-dev#7700

nicoddemus · 2020-09-09T13:12:43Z

@ssbarnea I've update the PR to include simple integration tests for pytest-cov, pytest-django, pytest-html, and pytest-sugar. I also fixed the issue you identified originally in pytest-cov.

We should backport the commit that adds the integration tests back to 6.0.x after this gets merged.

Fix pytest-dev/pytest-cov#430.

Probably that configuration is there for historical reasons, but doesn't make much sense to collect any *.py in testing as a potential test nowadays.

nicoddemus · 2020-09-12T02:03:18Z

I think this is a good starting point, but if others feel some plugin is missing, please speak up!

tox.ini

agronholm · 2020-09-13T11:56:07Z

You need to ensure that the tests are actually run. Add a failing case with strict xfail=True to make sure. I learned this the hard way when I found out AnyIO's hypothesis support had been silently broken the whole time.

src/_pytest/python.py

ssbarnea · 2020-09-13T12:09:01Z

You may want to look at the PYTEST_REQPASS feature added by pytest-plus which assures that an exact number of tests passed. That was aimed for CI/CD usage for avoiding accidents where tests were skipped.

graingert · 2020-09-13T12:09:07Z

tox.ini

+    pytest pytest_trio_integration.py
+    pytest pytest_anyio_integration.py


I think we should test all the async frameworks simultaneously - they have all agreed to use marks to allow them to all interoperate in a single test run, so we should test that here

At least as long as you don't use async fixtures. pytest-asyncio is known to appropriate those regardless of marks (see pytest-dev/pytest-asyncio#124 for the issue).

At least as long as you don't use async fixtures.

That's a very good point, we currently don't test any async fixtures here.

tox.ini

graingert · 2020-09-13T12:20:01Z

@ssbarnea what's the scope of this you're happy to accept? I think as plugin maintainers become aware of this PR more will want included.

I don't want to see a situation where this scope creeps all the way to testing all of pypi before it's merged! So I think we should merge this sooner and rather than later so maintainers can add their own smoke tests as PRs directly to pytest, rather than this branch?

graingert · 2020-09-13T12:23:14Z

installation of popular plugins does not break pytest.

how popular is popular? Can we nail down a criteria for inclusion?

altendky · 2020-09-13T14:14:18Z

You may want to look at the PYTEST_REQPASS feature added by pytest-plus which assures that an exact number of tests passed. That was aimed for CI/CD usage for avoiding accidents where tests were skipped.

Is that better than just requiring coverage in your tests?

altendky

The stated purpose is to test that pytest doesn't break. Shouldn't the full pytest suite be run instead of just a trivial case for each plugin?

tox.ini

ssbarnea · 2020-09-13T14:30:37Z

xdist and cov are ok dor start. good point about avoiding scope creep. we can always add more later.

tox.ini

nicoddemus · 2020-09-16T15:48:47Z

The stated purpose is to test that pytest doesn't break. Shouldn't the full pytest suite be run instead of just a trivial case for each plugin?

This makes sense, but I would rather start things slowly:

Plugins might interact with pytest differently in a way that breaks the pytest suite, but are not wrong: for example pytest-sugar changes output, and several tests check output and would break, requiring us to special case tests, or choosing a subset.
We can always improve things later the more we get the feeling how this will work. Having it being a quick sanity check run is beneficial, and reliable.

I don't want to see a situation where this scope creeps all the way to testing all of pypi before it's merged! So I think we should merge this sooner and rather than later so maintainers can add their own smoke tests as PRs directly to pytest, rather than this branch?

Agreed! This is already a large improvement over what we have today, so let's get this in as soon as we feel this is good enough.

nicoddemus · 2020-09-17T16:14:24Z

Add 3 more: pytest-flakes, pytest-qt and pytest-bdd. Also added pip check as suggested.

nicoddemus · 2020-09-17T16:15:39Z

Hmm as I feared: pytest-qt, besides depending on a large library, also requires an X server. While it is possible, I don't want to introduce complicated setups at this stage.

nicoddemus · 2020-09-19T14:23:30Z

Anything else? If not it would be nice we could get some approvals here and get this merged. 👍

hugovk

This is a great idea! Looks good. I agree on doing light testing of each plugin, this is a quick smoke test* to check nothing exploded. Should catch the major things, and plugins are still encouraged to test against pytest pre-releases.

(*Etymology: 'The term originates in hardware repair and has been applied to software. It's intended to be a quick test to see if the application "catches on fire" when run for the first time.')

hugovk · 2020-09-19T15:09:21Z

Oh, one thing to keep in mind: it's possible a new plugin release will break current master, and a new PR will start to fail because of the plugin release, and not the PR changes.

In such cases, it's a good idea to check if a master build also fails in the same way.

It may be needed to temporarily skip smoke testing that plugin to prevent all PRs failing, until either pytest or the plugin has a fix (because it could be a problem on either side, or both!).

But hopefully this would be rare. And in any case, it's good information to know there's a failure.

testing/plugins_integration/pytest_qt_integration.py

bluetech

As the person doing most of the breaking, this will be very helpful!

nicoddemus · 2020-09-19T18:56:12Z

Thanks everyone!

nicoddemus · 2020-09-19T18:56:42Z

Of course everyone feel free to add more plugins that they consider relevant. 👍

Co-authored-by: Bruno Oliveira <[email protected]> Co-authored-by: Thomas Grainger <[email protected]> Co-authored-by: Kyle Altendorf <[email protected]>

[6.0.x] Smoke tests for assorted plugins (#7721)

ssbarnea force-pushed the fix/plugins-impact branch from 579a78d to 7086e44 Compare September 5, 2020 15:02

ssbarnea mentioned this pull request Sep 5, 2020

pytest-cov is not tested with master branch of pytest pytest-dev/pytest-cov#430

Open

nicoddemus mentioned this pull request Sep 5, 2020

Release 6.0.2 #7718

Closed

ssbarnea and others added 2 commits September 9, 2020 10:08

New tox environment to test integration with some third part plugins

a6b5de7

This adds a new tox environment that tests that installation of popular plugins does not break pytest. We can add more as time passes.

Fix early_config.known_args_namespace not containing third-party options

0a27f69

Regression introduced in pytest-dev#7700

nicoddemus force-pushed the fix/plugins-impact branch from 7086e44 to 0a27f69 Compare September 9, 2020 13:10

nicoddemus changed the title ~~WIP: Test impact of plugin installation~~ Test impact of plugin installation Sep 9, 2020

nicoddemus marked this pull request as ready for review September 9, 2020 13:11

nicoddemus added 5 commits September 11, 2020 22:34

Force color in one run for pytest sugar to kick in CI

d39b45c

Add pytest-mock integration testing

83c648f

Add a README to testing/plugins_integration

59ee460

Add pytest-asyncio integration testing

be7c509

Add pytest-trio integration testing

5ce8c92

nicoddemus added the needs backport applied to PRs, indicates that it should be ported to the current bug-fix branch label Sep 12, 2020

Change python_files to only collect *.py in testing/python

0568fc7

Probably that configuration is there for historical reasons, but doesn't make much sense to collect any *.py in testing as a potential test nowadays.

nicoddemus changed the title ~~Test impact of plugin installation~~ Smoke tests for assorted plugins Sep 12, 2020

nicoddemus added 2 commits September 11, 2020 23:06

Force sugar output properly now on CI

222a120

Sort and match plugin deps and commands

b2795f8

nicoddemus mentioned this pull request Sep 12, 2020

doc: fix invalid url(socketserver.py) pytest-dev/pytest-xdist#599

Merged

2 tasks

graingert reviewed Sep 12, 2020

View reviewed changes

tox.ini Show resolved Hide resolved

add anyio smoke tests, and plugin recommendation

80a3345

graingert reviewed Sep 13, 2020

View reviewed changes

src/_pytest/python.py Show resolved Hide resolved

graingert reviewed Sep 13, 2020

View reviewed changes

tox.ini Outdated Show resolved Hide resolved

altendky reviewed Sep 13, 2020

View reviewed changes

tox.ini Outdated Show resolved Hide resolved

graingert reviewed Sep 13, 2020

View reviewed changes

tox.ini Show resolved Hide resolved

graingert reviewed Sep 13, 2020

View reviewed changes

tox.ini Show resolved Hide resolved

nicoddemus added 2 commits September 17, 2020 12:46

Add pytest-qt and pytest-flakes

f0eeb2c

Add pytest-bdd

6be9f7c

nicoddemus added 2 commits September 17, 2020 13:16

Remove pytest-qt due to the dependency to a X server

780752b

Add pip_pre

5e322d6

graingert self-requested a review September 19, 2020 14:27

nicoddemus mentioned this pull request Sep 19, 2020

ImportPathMismatchError when on UNC path #7678

Closed

hugovk approved these changes Sep 19, 2020

View reviewed changes

altendky reviewed Sep 19, 2020

View reviewed changes

testing/plugins_integration/pytest_qt_integration.py Outdated Show resolved Hide resolved

Add pytest-twisted smoke test

08de6dd

bluetech approved these changes Sep 19, 2020

View reviewed changes

graingert approved these changes Sep 19, 2020

View reviewed changes

nicoddemus added 2 commits September 19, 2020 15:37

Remove obsolete pytest_qt_integration.py

d75da3d

Fix linting and cleanup pytest_twisted_integration a bit

ba61962

nicoddemus merged commit b031a7c into pytest-dev:master Sep 19, 2020

nicoddemus added a commit that referenced this pull request Sep 19, 2020

Merge pull request #7775 from nicoddemus/backport-7721

89d03b2

[6.0.x] Smoke tests for assorted plugins (#7721)

hugovk mentioned this pull request Sep 27, 2020

Put smoke test deps in requirements.txt for Dependabot #7806

Merged

		pytest pytest_trio_integration.py
		pytest pytest_anyio_integration.py

Uh oh!

Smoke tests for assorted plugins #7721

Smoke tests for assorted plugins #7721

Uh oh!

Conversation

ssbarnea commented Sep 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicoddemus commented Sep 9, 2020

Uh oh!

nicoddemus commented Sep 12, 2020

Uh oh!

Uh oh!

agronholm commented Sep 13, 2020

Uh oh!

Uh oh!

ssbarnea commented Sep 13, 2020

Uh oh!

graingert Sep 13, 2020

Choose a reason for hiding this comment

Uh oh!

agronholm Sep 13, 2020

Choose a reason for hiding this comment

Uh oh!

graingert Sep 13, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

graingert commented Sep 13, 2020

Uh oh!

graingert commented Sep 13, 2020

Uh oh!

altendky commented Sep 13, 2020

Uh oh!

altendky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ssbarnea commented Sep 13, 2020

Uh oh!

Uh oh!

Uh oh!

nicoddemus commented Sep 16, 2020

Uh oh!

nicoddemus commented Sep 17, 2020

Uh oh!

nicoddemus commented Sep 17, 2020

Uh oh!

nicoddemus commented Sep 19, 2020

Uh oh!

hugovk left a comment

Choose a reason for hiding this comment

Uh oh!

hugovk commented Sep 19, 2020

Uh oh!

Uh oh!

bluetech left a comment

Choose a reason for hiding this comment

Uh oh!

nicoddemus commented Sep 19, 2020

Uh oh!

nicoddemus commented Sep 19, 2020

Uh oh!

Uh oh!

ssbarnea commented Sep 5, 2020 •

edited

Loading