-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Browserstack is not reporting errors properly #3256
Comments
The update job method is called from the runner. The runner keeps track if the tests are passing and updates the job if the updateJob exists on the driver provider. Taking a look at the BrowserStack api, I believe we are sending the correct payload for "Changing the status of a session". Also from the log message it does appear that the browser stack update job is being called. A possible test (since I am not familiar with BrowserStack) would be to console log if these failed tests are passing by editing the
|
@bvasilchik Any updates on this? |
@cnishina I apologize I haven't had time lately to test this. I did update the raw logs from browserstack in my original post |
Unfortunately, the logs that were posted appears to be just selenium server request / responses. Did you try the suggested logging above from my previous post? |
@cnishina I was just wondering if you've had any chance to review since I tried your suggested code. Or if you had any other ideas. Sorry it took me so long initially to respond Alright I had a chance to do what you suggested. I ran some tests which I knew would fail b/c they need to be updated to see what my results would be. I have included an image of what browserstack is reporting and the console logs of the failed tests that has the code for debugging that you mentioned to add. it seems every test has the following after failing
|
Looking at your console log messages, it appears that your tests are failing to find window.angular. So this would mean 1. it launched a browser window, 2. that the error is being properly reported. So now, why is it not finding window.angular? If this was working 100% fine on SauceLabs, and fine on your own machine, this would leave me to believe that this is a BrowserStack issue. |
those tests were going to fail b/c they were outdated so I knew they would all fail so that I could see what kind of results would be shown on browserstack. I figured it was the best way to see where some tests it does show the error but others are marked green. They should all show errors. So I was expecting them to fail and I would expect them to show as failing on browser stack as well. I guess I'm wondering what I do from here. This is what they told me. But i'm not sure what to tell them. It would be nice to have the correct results represented. I do thank you for all the help and support.
|
So I was very excited about this fix but it seems we are still getting failures that are marked with the green check-mark completed in browserstack. we are currently using 4.0.2 which I believe is supposed to have the fix. If there is anything I can do to help further troubleshoot. I know I was asked to add some debug information in the past, which I'm open to doing. Thank you for the work put in trying to resolve this issue. |
+1 |
I have recently switched from using saucelabs to using browserstack and noticed that some of the tests will report as failed via the console output but are not marked as failed on browserstack. I didn't notice this problem with saucelabs. I mentioned the problem to the browserstack support and they have told me that it's not a browsertstack problem but a protractor problem. This doesn't happen all the time but enough that I have noticed it. This failure doesn't always happen b/c it's intermittent and we have a script that re-runs any failed tests to make sure there aren't any false failures.
This is what they have told me
Bug report
v5.4.1
3.3.0
Chrome 50.0
OS X Yosemite
Here are the raw logs from browserstack
The text was updated successfully, but these errors were encountered: