Feature: Distinct Junit-Results with multiCapabilities

[zakrhol]

When setting up multiCapabilities and adding a jasmine-junitReporter, then specs running for all capabilities produces the same junit-result.xml file, where one replaces the other.

I have read in #60 that it is possible to change the filename of the result for each capability. But as far as I have tried, that is not always sufficient.

• Although the name of the junit-xml was changed, the name of the test-suites and test-classes remain equal and a CI-System like Jenkins is not able to display the results properly. I have tested this: when jenkins faces the same suite-name again, it is simply ignored - if the first capability passes and the second not, the test shows as passed. This may be a bug in Jenkins, but even if it shows all tests I cannot see in which capability it has failed.
• Changing the name by adding browsername and version as shown in #60 is also not enough for my cases: I always use the same browser but only change browser size to test for Table/SmartPhone resolutions (Bootstrap). Unfortunately browser.getCapabilities(); does not contain something useful (and readable) to distinguish the capabilities.

To get it working properly I made the following changes and I want to ask whether that makes sense or can be solved better.
I give each capability a name. To access the capabilities in the onPrepare function (browser.getCapabilities(); only has driver/browser stuff), I changed runner.js

Runner.prototype.runFilenamesOrFns_ = function(source) {
...
if (typeof filenameOrFn === 'function') {
        filenameOrFn();

to

Runner.prototype.runFilenamesOrFns_ = function(source) {
...
if (typeof filenameOrFn === 'function') {
        filenameOrFn(this.config_);

In the onPrepare the capability-name is appended to the suite and spec name and also the junit.xml filename.
My config.js now looks like this (simplified):

specs: [
      'test/protractorTests/all/**/*.js'
  ],

  multiCapabilities: [{
        'browserName': 'chrome',
        'chromeOptions' : {
            args: ['--lang=en',
                   '--window-size=800,800']
        },
        name: '[md]',
        specs: 'test/protractorTests/md/**/*.js'
      },{
        'browserName': 'chrome',
        'chromeOptions' : {
            args: ['--lang=en',
                   '--window-size=350,650']
        },
        name: '[xs]',
        specs: 'test/protractorTests/xs/**/*.js'
      }],
  onPrepare: function(config_) {
        require('jasmine-reporters');
        var capsPromise = browser.getCapabilities(); //does not contain useful stuff
        capsPromise.then(function(caps){
            var browserName = caps.caps_.browserName.toUpperCase();
            var name = config_.capabilities.name; //get the unique name
            var prePendStr = browserName + "-" + name + "-";
            var junitReporter = new jasmine.JUnitXmlReporter("build/test-results", true, true, prePendStr);
            junitReporter.reportSpecResults = (function(){
                junitReporter.superReportFct = junitReporter.reportSpecResults;
                return function(spec){
                    if (spec.description.indexOf(name) === -1){
                        spec.description = spec.description + name;
                    }
                    junitReporter.superReportFct(spec);
                }
            })();
            junitReporter.getFullName = (function(){
                junitReporter.supergetFullName = junitReporter.getFullName;
                return function(suite){
                    return junitReporter.supergetFullName(suite)+name;
                }

            })();
            jasmine.getEnv().addReporter(junitReporter);
        });

        var fs = require('fs');
        jasmine.getEnv().addReporter(new function() {
              this.reportSpecResults = function(spec) {
                if (!spec.results().passed()) {
                  //take screenshot
                    browser.takeScreenshot().then(function(png) {
                          var stream = fs.createWriteStream("failure_"+spec.description+".png");
                          stream.write(new Buffer(png, 'base64'));
                          stream.end();
                        });
                }
              };
            });
      },

This automatically solves the same issue with the failure-screenshot's filename, where also one replaces the other. (because the junit-reporter changes the specs description permanently - not sure if this has other impacts)
Having the config_ (capability&spec) available in onprepare might also be helpful in other use-cases, wouldn't it?
I think there could be a build-in way protractor provides to create unique test-results - or am I missing some better way?

[psech]

I am facing the same problem.
Long time ago (before multiCapabilities) I was using many configs to describe browsers.
To running tests I was using grunt-protractor-runner and for sending them to Grid in parallel I was using grunt-concurrent.

chrome: {
    options: {
        configFile: "configs/chromeConf.js",
        args: {
            params: 'chrome_24'
            }
      }
},

Then I was using provided param to name report file. It was complicated.

Now I am glad I can use multiCapabilities and send test to the Grid in simple way, but there is a problem with getting reports for each browser.
It would be perfect to have one, concatenated xml file in jUnit format to use in Jenkins.

Is there any chance that Protractor will take care about that on its own?

Btw. @zakrhol thanks for the tip, I will try this ;)

[alecxe]

+1 This would be really good to have.

[danopia]

Any progress on this in the past 9 months? I'd be cool with one XML file per browser.

[reppners]

I provided a pull-request for the jasmine-reporters package larrymyers/jasmine-reporters#101

With this it is possible to use the current capability properties to define a unique suite name that will also be reflected as a filename. So the result is one xml per capability.

[spengilley]

Is this still outstanding? or has this capability been added? Can this issue be closed now?

[juliemr]

I'm not sure there's a concrete request for Protractor here, so closing this issue. Please open up a new one if there's something specific and in-scope for Protractor to add.