Add common final check to every test.

[choppsv1]

tl;dr -- Is there a way to hook into each test run just after the test has run but in a place where I can still call pytest.fail() and have the right thing happen?

We have a testing framework built upon pytest for FRR. A custom option we added was --valgrind-memleaks which causes all the FRR daemons to be run under valgrind. I've written a check that looks for new valgrind files generated from memleaks. I'd really like to fail the test when this check finds new memleaks files so that we see that this test in particular caused a memleak. Ideally I'd just use a fixture, yield to the test and then do the check after, but this will cause a pytest error instead of a test failure -- which then won't be logged/collected/noted by all the CI infrastructure etc which is looking for test failures.

The code (broken) currently tries to use and catch a pytest.fail() from inside a try/except statement to try and construct a excinfo to plug into the call.excinfo in a make report hook but this doesn't seem to work right and seems hackish anyway.

[nicoddemus]

Hi @choppsv1,

I think you will need a custom plugin that implements pytest_runtest_call to call item.runtest(). It should check for any leaks, and call pytest.fail accordingly. Something like:

# content of conftest.py

def pytest_runtest_call(item: pytest.Item) -> None:
    with capture_mem_leaks() as leaks:
        item.runtest()
    if leaks:
        pytest.fail(f"test {item.name} leaked memory: {leaks}")

[choppsv1]

I tried this (well w/o the with syntax) and have an issue that the tests run twice once by my confitest.py:pytest_runtest_call() and once by _pytest.runner.py::pytest_runtest_call()

[choppsv1]

I figured it out, use hookwrapper:

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item: pytest.Item) -> None:
    # stuff I did before that should execute before test.

    # Let the default pytest_runtest_call execute the test function
    yield

    check_for_memleaks()