chore(ci_visibility): move retry logic to pytest_runtest_protocol

[vitor-de-araujo]

This is essentially #13376, recreated now that the fix in the unittest suite (#13445) has been merged, and with the previous pytest_runtest_protocol logic wrapped in a try/except to avoid breaking the pipeline in case of an internal error in CI Visibility.

---

Currently we let pytest's builtin pytest_runtest_protocol hook to run the test, and we check whether to retry at the makereport stage. This has a number of consequences:

• We don't have access to the setup, call, teardown reports all at once; we see one at a time during makereport, and we have to patch them and stash information around to keep state across each time we pass through the reports for a given test.
• pytest logs each report as it is created, so we have to patch them at the right time so they get printed correctly (to e.g. change the outcome from FAILED to ATR INITIAL ATTEMPT FAILED; this affects not only printing, but also the session exit status).
• In particular, for EFD when the initial attempt passes, we run too late and the PASSED status was already logged by the time we patch it, so it shows PASSED instead of EFD INITIAL ATTEMPT PASSED. Not only that, but we have to handle this case specially when generating the terminal summary.

This PR moves the retry logic from the pytest_runtest_makereport hook to the pytest_runtest_protocol hook. This means we replace pytest's own pytest_runtest_protocol with our own. We invoke pytest's internal runtestprotocol function directly from our hook, so the behavior of our hook is similar to the pytest's own hook. The difference is that we call this function with log=False, so pytest doesn't log the setup, call, teardown reports as they are created. Instead, we collect all reports, patch them as needed, and then print them out. This means we can write the logic having full knowledge of the final status of a test run, instead of patching things as we see them during setup, call and teardown.

For retries, the responsibility for logging the statuses is moved to the retry handlers themselves, so they can delay printing to after the reports have been patched. In principle, we could even decide to not print the retry results individually and only print the final status (which would make for a cleaner output), but this can come in a future version.

For EFD, the special final outcomes (dd_efd_final_passed, etc.) are replaced with plain passed, failed, skipped states, which xdist can handle, and the final states are only used in efd_get_teststatus (called from the pytest_report_teststatus hook).

Future work:

• Attempt to Fix has to be modified in similar ways to EFD, but it also has to handle quarantine, so it's a bit more involved.
• xdist still prints EFD INITIAL ATTEMPT for all atempts (not just the first one).
• The whole retry logic outside of pytest should be refactored (see chore(ci_visibility): refactor test retry logic #13224), but this PR is a first step to make the rest possible.

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

ddtrace/contrib/internal/pytest/_atr_utils.py                           @DataDog/ci-app-libraries
ddtrace/contrib/internal/pytest/_attempt_to_fix.py                      @DataDog/ci-app-libraries
ddtrace/contrib/internal/pytest/_efd_utils.py                           @DataDog/ci-app-libraries
ddtrace/contrib/internal/pytest/_plugin_v2.py                           @DataDog/ci-app-libraries
ddtrace/contrib/internal/pytest/_retry_utils.py                         @DataDog/ci-app-libraries
ddtrace/contrib/internal/pytest/_utils.py                               @DataDog/ci-app-libraries
ddtrace/contrib/internal/pytest/plugin.py                               @DataDog/ci-app-libraries

[github-actions]

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 248 ± 4 ms.

The average import time from base is: 248 ± 5 ms.

The import time difference between this PR and base is: -0.4 ± 0.2 ms.

The difference is not statistically significant (z = -1.99).

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 1.869 ms (0.75%)

ddtrace.bootstrap.sitecustomize 1.191 ms (0.48%)

ddtrace.bootstrap.preload 1.191 ms (0.48%)

ddtrace.internal.remoteconfig.client 0.582 ms (0.23%)

ddtrace 0.678 ms (0.27%)

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-05-19 15:47:00

Comparing candidate commit dcc238e in PR branch vitor-de-araujo/SDTEST-1850/refactor-retry-logic-2 with baseline commit 1219e33 in branch main.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 495 metrics, 8 unstable metrics.

scenario:iastdjangostartup-iast

• 🟩 execution_time [-389.081ms; -173.970ms] or [-16.350%; -7.311%]