Can the junitxml module give a better error when out of memory?

[j9ac9k]

Greetings from pyqtgraph! Huge fans of pytest, hope everyone here is doing well. I'm opening an issue here to solicit feedback, and make myself available for troubleshooting/testing.

Sorry for the wall of text, this discussion tab is created after a series of frustrating results, and by chance stumbling across a workaround.

Scenario:

A PR adding some new feature was submitted (👋🏻 @ntjess). PyQtGraph has 18 different test pipelines on GH Actions (all 3 OSs, different python and qt-binding versions). On one pipeline, our test suite would bail midway through, with exit code 3 (CI run). This failure is consistent, I can re-run the pipeline 100 times, and this same failure on this pipeline would occur each time. The wrinkle here is that nobody could reproduce locally on their machine.

An oddity here is that the error of the test failed was not related to the PR. The PR specific tests were next in the test sequence. Due to the nature of GUI testing in a headless environments, we do see periodic errors like this, but usually their more intermittent, and spread out across different pipelines. After a very through review of the PR, and commenting out/re-enabling different positions of it, we realized the cause of the PR wasn't necessarily within the code diff, but this code diff was bringing out an existing issue.

The next step was for me to rerun the test suite to skip the test that actually failed, so I replaced

pytest tests -v  --junitxml pytest.xml

with

pytest -v -s -k "not test_nan_image" 

This passed...eventually I realized the only difference was the --junitxml pytest.xml argument (this is a hold-over from a feature we made use of azure pipelines, but we don't use it any more so removing it wasn't a big deal for us).

Doing a search in the issue tracker, #3328 caught my attention, and then I realized we might be running out of memory (on the very limited about of memory we have in CI) and that's why our test suite is failing/erroring.

So along that lines, this discussion is a bit of a "bug report" in the sense of junitxml should probably not result in outright failure, and if it does result in outright failure, a more helpful message might be worthwhile (feature request).

I'm sorry I don't have a MWE I can share with you. I can only share some branches where the test suite pass and fail, where that's the only diff between the two.

• Failing Branch
• Passing Branch

I'm happy to do some testing/experimentation if it would be helpful.

Thanks you your work on pytest, we're big fans!

[nicoddemus]

Thanks @j9ac9k for the detailed report, we appreciate it!

@RonnyPfannschmidt already did some work in the past to reduce junitxml's memory footprint, one of the things it did was avoid unnecessary Python structures in memory, keeping only the final string which is written to the file at the end of the session. Do you have access to the junitxml file? How large is it?

One improvement that could be done is to write the XML contents to a temporary file, and only move it to the final location when the session ends. This would allow us to not keep the entire string in-memory.

[j9ac9k]

Do you have access to the junitxml file? How large is it?

I do not, but I'll set some time aside later to upload the file as an artifact on the failing branch.

[j9ac9k]

So, slight wrinkle ... there appears to be no pytest.xml file when pytest fails in this fashion, so there was no artifact to upload:

https://github.com/j9ac9k/pyqtgraph/runs/3185483563?check_suite_focus=true

I'm going to run the test suite on my local windows machine in a similarly configured environment and follow up shortly.

[j9ac9k]

I ran the test suite locally on my windows machine, the pytest.xml file was 20 KB, I've uploaded it here (I had to rename the extension from xml to zip so github would let me attach it).

Let me know if there is any other useful information I can provide.

pytest.zip

[nicoddemus]

Thanks!

But the XML from the other successful runs would suffice, I wanted to know the final XML size.

But only 20kb? Weird, then definitely the problem is not the in-memory file size... 🤔

[j9ac9k]

Actually, it appears it might even be smaller than that; not sure how to get to artifacts on github actions, but on the artifact upload task, it says the file size is 2915 bytes

https://github.com/j9ac9k/pyqtgraph/runs/3186100967?check_suite_focus=true