Testing pytest plugin I'm building, how to introspect that Item has an attribute added (using pytester/testdir)?

[joshua-badger]

TL;DR Is there a way, using testdir/pytester to introspect an Item after it has been run (for testing a plugin, not implementing a plugin)?

I'm trying to write a plugin for our organization where we standardize saving test artifacts and displaying them in a pytest-html report.

In the test file, I use testdir.inline_genitems() on the files I build in testdir (see below; have omitted pieces of code irrelevant to the question). From that, I'm able to get the items' nodeids and validate that I've copied the artifacts to where I expect them. Awesome.

But from a code coverage point of view, I don't have a good scope for whether my test items are correctly recording. I'd love to be able to introspect them after a run. (Specifically, whether the artifacts were successfully gathered or not, which would be recorded in the Artifact class.) Unfortunately, I cannot figure out how to do so. Not awesome.

For example: I've tried the runtest() method on the Items I got from inline_genitems() (with or without the lines. It throws the following error:

>   testargs = {arg: funcargs[arg] for arg in pyfuncitem._fixtureinfo.argnames}
E   KeyError: 'some_value'

/path/to/project/.venv/lib/python3.6/site-packages/_pytest/python.py:179: KeyError

Is there something obvious I've missed? Is there a better way to set up my test file (for example, by NOT using the inspect module to copy hooks/fixtures defined by my plugin into a conftest.py file)?

Relevant code below:

# contents of test_plugin.py

@pytest.fixture(autouse=True)
def built_conftest(testdir):
    conftest_src = inspect.getsource(the_plugin)
    testdir.makeconftest(conftest_src)
    yield


@pytest.fixture
def collectable_artifacts_test_source(artifact_fp):
    return f"""
        import pytest

        @pytest.mark.parametrize("some_value", list(range(3)))
        def test_a_new_thing_with_artifacts(some_value, artifact_collection):
            artifact_fp = "{artifact_fp}".format(some_value)
            with open(artifact_fp, "w") as fo:
                fo.write("Some silly nonsense")
            artifact_collection(artifact_fp, f"Artifact {{some_value}}")
        """


@pytest.fixture
def collectable_artifact_test(testdir, collectable_artifacts_test_source):
    # Build mock test file, with what should be unique nodeids for each test item
    test_a_thing_file = testdir.makepyfile(test_collectable=collectable_artifacts_test_source)
    return test_a_thing_file


def test_artifact_dumper_successfully_collects(
    testdir,
    collectable_artifact_test,
    non_collectable_artifact_test,
    other_artifact_location,
    artifact_template_filename,
    non_artifact_filename
):
    """
    Pass Criteria:
        - Only artifacts gathered with artifact_collection fixture are collected to artifact directory
        - Collected artifacts are grouped by test node id, so they are separated easily
    Steps:
        1. Generate all the items expected to be run so we can get the nodeids
        2. Get node ids from generated items, build out expected/non-expected artifact filepaths
        3. Run the tests, verify successful test run
        4. Verify expected artifacts have been moved to expected locations
        5. Verify unexpected artifacts do not exist
    """

    # 1. Generate all the items expected to be run so we can get the nodeids
    tests_to_run, _ = testdir.inline_genitems(
        collectable_artifact_test, non_collectable_artifact_test
    )
    # assert len(tests_to_run) == 4

    # 2. Get node ids from generated items, build out expected/non-expected artifact filepaths
    *has_artifact_nodeids, no_artifact_nodeid = [i.nodeid for i in tests_to_run]
    artifact_dir = os.path.join(
        other_artifact_location, the_plugin.TEST_ARTIFACTS_INTERMEDIATE_DIRPATH
    )
    expected_final_artifact_locations = [
        os.path.join(
            artifact_dir,
            nodeid,
            artifact_template_filename
        ).format(i) for i, nodeid in enumerate(has_artifact_nodeids)
    ]
    not_expected_final_artifact_location = os.path.join(
        artifact_dir,
        no_artifact_nodeid,
        non_artifact_filename
    )

    # 3. Run the tests, verify successful test run
    result = testdir.runpytest()
    assert result.ret == 0

    # FIXME: Trying desperately to run the items and introspect that they have a ArtifactCollection
    #        as a property!
    # for test_item in tests_to_run:
    #     test_item.runtest()
    #     assert getattr(test_item, "test_artifacts", None)

    # 4. Verify expected artifacts have been moved to exxpected locations
    for expected_final_artifact_location in expected_final_artifact_locations:
        assert os.path.exists(expected_final_artifact_location)

    # 5. Verify unexpected artifacts do not exist
    assert not os.path.exists(not_expected_final_artifact_location)

# relevant contents of the_plugin.py

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    """
    Hook adding convenience properties to the node attribute of request fixture objects.
    Allows for quick introspecting of results
    """
    logger.info("Prepared to gather any test artifacts on failure")
    item.test_artifacts = ArtifactCollection(item)
    # execute all other hooks to obtain the report object
    outcome = yield
    rep = outcome.get_result()

    # set a report attribute for each phase of a call, which can
    # be "setup", "call", "teardown"

    setattr(item, f"rep_{rep.when}", rep)

@pytest.fixture
def artifact_collection(request):
    """
    Callable fixture to dump specific, desired artifacts to a known location so they can be
    linked to in the HTML report
    """
    test_artifact_dump_path = os.path.join(TEST_ARTIFACTS_DIRPATH, request.node.nodeid)

    def _dump_artifact_to_dump_path(original_artifact_filepath, artifact_name):
        os.makedirs(test_artifact_dump_path, exist_ok=True)
        artifact_filename = os.path.basename(original_artifact_filepath)
        dumped_filepath = os.path.join(test_artifact_dump_path, artifact_filename)
        logger.info(f"Taking artifact and moving it from {original_artifact_filepath} "
                    f"to {dumped_filepath}")
        try:
            os.rename(original_artifact_filepath, dumped_filepath)
        except FileNotFoundError as e:
            logger.error(f"Couldn't find expected artifact at {original_artifact_filepath}!")
            request.node.test_artifacts.report_uncollectable_artifact(
                original_artifact_filepath,
                artifact_name,
                reason=f"Couldn't find expected artifact at {original_artifact_filepath}!"
            )
        except Exception as e:
            logger.exception(
                f"Couldn't move artifact from {original_artifact_filepath}, see traceback!"
            )
            request.node.test_artifacts.report_uncollectable_artifact(
                original_artifact_filepath, artifact_name, reason=repr(e)
            )
        else:
            logger.info(f"Artifact saved to {dumped_filepath}")
            request.node.test_artifacts.collect_artifact(dumped_filepath, artifact_name)

    return _dump_artifact_to_dump_path


class ArtifactCollection:
    """Collection of artifacts for failed tests"""

    def __init__(self, test_node):
        self._test_node = test_node
        self.artifacts = []

    def __iter__(self):
        return iter(self.artifacts)

    def collect_artifact(self, artifact_filepath, name_for_report):
        real_path = os.path.abspath(artifact_filepath)
        self.artifacts.append(
            Artifact(real_path, name_for_report, collected=True)
        )

    def report_uncollectable_artifact(self, original_filepath, name_for_report, reason=""):
        self.artifacts.append(
            Artifact(original_filepath, name_for_report, collected=False, reason=reason)
        )


class Artifact:
    """Artifact object for attaching to the HTML report"""

    def __init__(self, artifact_filepath, name_for_report, collected=True, reason=""):
        self.filepath = artifact_filepath
        self.name = name_for_report
        self.collected = collected
        self.uncollected_reason = reason or None

[bluetech]

I haven't read the question fully, but just a clarifying question - are you using testdir for testing your plugin, or for implementing it?

[joshua-badger]

Testing. Definitely testing.

[joshua-badger]

Basically, I want to validate that once an item is run, the ArtifactCollection has been populated with the expected Artifacts so that I can set up the extra column in the HTML report.

[RonnyPfannschmidt]

Unfortunately pytest primarily has tools for full blown system tests, not so much smaller functional test components

[joshua-badger]

Okay. Perhaps I'll see if I can wrap some mocks around the ArtifactCollection methods and change my strategy, then. Thanks!