Support alternative xUnit (JUnit) formats

[mverma-va]

Please add support for creating JUnit xml output report, we need this badly. Since most other tool support that.
Like right now we have pybot -x for generating xUnit xml format report files.
Similarly plz add support to generate JUnit xml format report file.

Or maybe point me to some script which will convert robot output.xml to JUnit format (http://llg.cubic.org/docs/junit/)

[pekkaklarck]

How would that jUnit output be different to the current xUnit output?

[pekkaklarck]

Related to my previous comment, have you tested using the xUnit output?

See also https://en.wikipedia.org/wiki/XUnit.

[mverma-va]

Yes I have tried the xUnit output using the pybot -x option. One difference is that the xunit report file is missing the top level "testsuites" node. But you might have seen the Junit report file begins with "testsuites" node.
We are using couple of other tools which gives output in stantard Junit format, so it would be nice if we could get a similar junit format output from pybot.
I am new to this, so not sure how to fix this

[pekkaklarck]

The problem is that there is no such thing as the standard jUnit format. This is highly annoying.

The current format used by RF is based on some jUnit output variation, but there could be others. If you can find statistics that other formats are used a lot more widely, we can consider changing the current format. Alternatively you can create a script that converts the current format to a format understood by tools you use. I don't like the idea of us adding multiple xUnit formats when nobody has bothered to create an official standard.

[mverma-va]

ok thanks, will workaround my creating our own custom output. This bug can be closed now.

[pekkaklarck]

After thinking this a little bit more, xUnit format configurable is not a totally impossible idea. I definitely don't want a totally new output, but having an option like --xunitformat to control the format would be OK. That would basically have two preconditions:

• Someone studies what xUnit XML flavors there are. Based on that we can design this functionality in more detail.
• Someone is interested enough about this to provide a pull request based on the agreed design.

Do you @mverma-va have interest to look at this?

[pekkaklarck]

Anyone interested to add support for xUnit file configuration? If not, I'll close this issue.

[davidk1]

We are using Bamboo and Robot FW. There is only JUnit parser in Bamboo so we'd appreciate to have JUnit output from Robot FW.

[pekkaklarck]

@davidk1: Robot already provides JUnit output, we just use more general xUnit name for it. The problem is that there is no official standard for that format and Robot only implements one variant. I'm fine making xUnit format configurable. See my comment above for what it would require.

[pekkaklarck]

I consider this a valid enhancement and set labels accordingly. Actually getting this implemented requires someone providing a pull request or paying me or someone else for development.

[pekkaklarck]

See #442 for discussion about adding the original xUnit support.

[mverma-va]

Just adding for reference that soapui junit xml report is really well formatted and includes most of the required info.

[pekkaklarck]

It would be great if people using xUnit/jUnit outputs would provide a little more information about the current situation about the output "standard":

• List in comments tools that work with Robot's current --xunit outputs.
• Even more importantly, list tools that don't work with the current --xunit outputs.
• Add references to specs about the output format if you know any.
• Add references to outputs files produced by other tools. Adding an attachment is fine too.

[drew-royster]

I'm wanting the same thing as @mverma-va. I've been using junit-viewer to view my test outputs, but it doesn't appear to be working with robot generated xunit xml files.

I'd love to have this functionality

[trogdor259]

List in comments tools that work with Robot's current --xunit outputs.

I've been using Bamboo's JUnit Parser to get my results listed and that works great.

[pekkaklarck]

@drew-royster: Could test why junit-viewer doesn't work with Robot's xUnit outputs? It would be great if you could try manually changing outputs to make them compatible so that we'd know what kind of changes would be needed.

[pekkaklarck]

It would be great if someone could study what's the current status with different xUnit/jUnit XML outputs. If there are different standards in wide use, we could make it somehow possible to use select which one to use. If there's a dominant standard that's different to what we currently use, we could even consider changing out xUnit output to use that. That could only happen in a major version, though.

Anyway, this needs external help from people familiar with this topic or interested to study that. I don't expect the actual implementation to be too complicated.

[KishoreNateshOfficial]

The xunit report generated cannot be used in Jenkins Zephyr plugin to update test cases on Jira. Because it uses Junit report or might Xunit report format is not compatible with that plugin. :(

[pekkaklarck]

xUnit and jUnit reports should in theory be the same. The problem is that at least earlier (see #442) there has been no official spec for xUnit/jUnit compatible reports. If there is now, we can either change xUnit output or make it configurable somehow. Someone interested in this topic should study the current situation and report the findings here.

[KishoreNateshOfficial]

This is the issue and Jobprogress token might be missing in Xunit format.

[pekkaklarck]

I seriously doubt jUnit outputs have a standard attribute jobProgressToken. Related to that, what we'd need to get forward is somekind of jUnit output specification. Last time I checked such standard didn't exist but that was long time ago.

[rasjani]

OP's linked description of junit format looks about right - atleast in the form that it can be digested by most of the tooling (eg, jenkins). I have not really used "xUnit" format ever but my understanding is that there are difference (as there are also multiple versions?). As a side comment, even if there is no such thing as "standard" for junit, it would be acceptable if the format is something that CI tools with their report plugins can digest and if im not mistaken, atleast jenkins does have xsd schema defined so it could be used as a basis ?

Anyway, I would be interested on picking this up when i have downtime.

Somewhat related issues:

pytest-dev/pytest#3547
cpputest/cpputest#1163

And as i mentioned above, the xunit plugin provides xsd files here that could be used atleast for investigation why current xunit format is not suitable or as a basis for creating junit format:

https://github.com/jenkinsci/xunit-plugin/tree/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd

[pekkaklarck]

Thanks for your investigation @rasjani and hope you have time to look at this more closely at some point. It's interesting, but not really that surprising, to see that other tools are having the same problem. It's super annoying nobody standardizing jUnit/xUnit back in the early days. Do you know the origins of the format Jenkins uses? Jenkins is so ubiquitous that they standardizing to certain format is likely to make their format at least a de-facto standard.

[rasjani]

junit-10.xsd: doesnt validate "hello world" like xunit report produced by robot due to testsuite attribute "skip" not being allowed as it has been removed - only "skipped" is left.

junit-9.xsd and junit-8.xsd against the same helloworld xunit report passes validation just fine.

So, from atleast my pov, the only thing that should be done for now is to rename "skip" attribute into "skipped" because that should be backwards compatible. Also, at the time of this ticket was created, Pekka's assumption was indeed correct, there should not be huge differences between xunit & junit ;)

@KishoreNatesh i think your your issue most likely has nothing to do with the xunit format itself. Looking at the code (https://github.com/jenkinsci/zephyr-for-jira-test-management-plugin/blob/master/src/main/java/com/thed/zephyr/jenkins/utils/rest/TestCaseUtil.java) and the stacktrace you provided, the failure happens with the request response for from zephyr when it tries to assing a test to a cycle.

[pekkaklarck]

@rasjani Is that junit-10.xsd used by Jenkins nowadays? Can you test would it be enough to change skip to skipped in Robot's xunit outputs? The place where to do that is here:
https://github.com/robotframework/robotframework/blob/master/src/robot/reporting/xunitwriter.py#L54

[ewolz3]

@pekkaklarck It's used by Jenkins xUnit plugin. When I edit xunitwriter.py L54 and changed skip to skipped, Jenkins doesn't throw that error anymore.

Plugin's Schema Definitions

junit-10.xsd only allows skipped for testsuite: https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd#L164

junit-9.xsd allowed skip, skips, or skipped: https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-9.xsd#L111

Edit: Just to note, the existing Robot Framework xunit format works with the Jenkins built-in JUnit plugin, just not the xUnit plugin using the JUnit pattern.

[kanchi240]

Feed output.xml or xunit.xml to junitth(http://junitth.sourceforge.net) failed, junitth is great tool for reporting, it will be great if robot will support it.

JUnit format:
http://llg.cubic.org/docs/junit/
@pekkaklarck

update:
Add ‘time’ attribute into the test suite element of the xunit.xml file, then it is working now, should robot fix it?

[pekkaklarck]

@ewolz3 Did I get it right that if we change skip to skipped we get support for junit-10.xsd and nothing breaks? That would be a very simple change and apparently safe as well.

Could you @kanchi240 clarify what you mean with the time attribute? Have you tested does adding that cause problems with Jenkins or other tools processing xUnit/jUnit outptus?

[rasjani]

@pekkaklarck @kanchi240 regarding missing time attribute: https://issues.jenkins-ci.org/browse/JENKINS-51797 - it was reported as bug in jenkins side and was quickly fixed.

[pekkaklarck]

Did I get these points about the time attribute right?

• [xj]Unit format supports optional time attribute in testsuite elements.
• Robot doesn't add such attribute (but it adds to testcase elements).
• Jenkins had a bug that time attribute wasn't optional but that's now fixed.

If I got all that right, then we don't need to do anything, but I don't see problems us adding the time attribute to testsuite elements either. If @kanchi240 or someone else considers that to be a good idea, please submit an issue about it. Although that probably is a a safe addition, it's still best done in a major release and now when RF 3.1 is under development would be a good time to add it.

[pekkaklarck]

Have I got these points about Jenkins compatibility right?

• The problem is that we use skip, not skipped, attribute with testsuite elements.
• Changing skip to skipped doesn't break any other tool using [xj]Unit outputs.

If the above points are correct, then we don't need separate jUnit output support nor do we need to make the xUnit output we produce configurable. All we need is simply changing skip to skipped. If that's the case, then we need to update the title of this issue or possibly submit a new one about skip -> skipped change and close this one. Regardless of that, we'd also need a PR with code and test changes. Shouldn't be a big task.

[pekkaklarck]

By the way, is it a problem that Robot's xUnit output always only contains exactly one testsuite and all tests are inside it? This happens even if you are executing a test suite structure with multiple nested suites. Does [xj]Unit output in general support such nested structure? If that's supported, I'd be fine changing also that in RF 3.1.

[rasjani]

I did some experimentation with Jenkins addons and here's the findings:

• JUnit Plugin 1.26.1

  • Works with both "skip" && "skipped".

• xUnit plugin 2.3.0 with "Junit" format:

  • Works Only with "skipped" due to recent changes.

I've provided a patch that fixes this issue..

Also, worth to empasize that when using "xunit" addon, you have to pick "JUnit" format, nothing else will work and doesn't even make sense to support..

[pekkaklarck]

Thanks @rasjani for investigation! I'll add this RF 3.1 scope and since you apparently have a PR ready I target this to beta 1 that's planned for next Wednesday.

PS: The list in your comment may not be correctly formatted.

[pekkaklarck]

Or should we actually submit a new issue about skip -> skipped? This issue is about configuring xunit outputs and it seems that's not needed.

[pekkaklarck]

Thanks to @rasjani we now got issue #2978 about changing skip to skipped and PR #2977 to actually implement that. It seems we don't need any separate configuration to control the xUnit format so this issue can be closed.

As discussed above, we could also consider adding <testsuite> elements for each test suite, not only to the top-level suite as currently and we could also add time attribute to the <testsuite> elements. These also require separate issues.

[lightingbijoy]

 DATADOG_API_KEY=api_key DD_ENV=ci datadog-ci junit upload \  ✔  282  09:17:31
--service robot-framework
execution_reports/
Starting upload with concurrency 20.
Will look for jUnit XML files in execution_reports/
service: robot-framework
❌ Invalid jUnitXML file [execution_reports/output-20220208-110758.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-110904.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-121124.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-150334.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-152001.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-152303.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-164031.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-164226.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-183019.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-183319.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-232227.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-233347.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-233437.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-233627.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-234226.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220208-234254.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220209-000937.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220209-001044.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220209-004704.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220209-004739.xml]: Neither nor are the root tag.
❌ Invalid jUnitXML file [execution_reports/output-20220209-004851.xml]: Neither nor are the root tag.

I am getting this error as the parser is looking for testsuites or testsuite in the output xml. How can I fix this?