-
-
Notifications
You must be signed in to change notification settings - Fork 22
attempt number is added to screenshot name, causing retries of failures to inappropriately succeed #168
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
closes #168 Signed-off-by: Jakub Freisler <[email protected]>
🎉 This issue has been resolved in version 3.1.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Hey @froodian, First of all - thank you for such detailed issue! ❤️ I never have used Cypress retries option, so you description made it much, much easier for me to understand what's the issue and how it can be covered. Please try the new release 3.1.0 and come back with a feedback (simple "it works!" is enough 😄 ) |
# [1.1.0](v1.0.2...v1.1.0) (2022-11-03) ### Bug Fixes * **deps:** update all non-major dependencies ([FRSOURCE#165](https://github.com/braze-inc/cypress-plugin-visual-regression-diff/issues/165)) ([602640f](602640f)) * ts declaration generation ([1a1e0cc](1a1e0cc)) * typings for older typescript ([FRSOURCE#170](https://github.com/braze-inc/cypress-plugin-visual-regression-diff/issues/170)) ([96499ec](96499ec)), closes [FRSOURCE#167](https://github.com/braze-inc/cypress-plugin-visual-regression-diff/issues/167) ### Features * support Cypress retries functionality ([FRSOURCE#171](https://github.com/braze-inc/cypress-plugin-visual-regression-diff/issues/171)) ([7d7d010](7d7d010)), closes [FRSOURCE#168](https://github.com/braze-inc/cypress-plugin-visual-regression-diff/issues/168)
Hi @FRSgit - thanks for the fast fix! It looks like with 3.1.0, it mostly works - but, if the first attempt fails for unrelated reasons (before the screenshot), the retry attempt creates I think this could probably be fixed by making the code a little more like
to make sure |
Hey @froodian, You're right - this condition is not enough. But after giving it a though solution proposed by you (and the implementation for I need to think about it for a moment, if you have any ideas - don't hesitate to post them here |
🎉 This issue has been resolved in version 3.1.1 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Just released 3.1.1 which should handle all of the cases above correctly. Please give it a go 🙏 |
Yes, I see what you're saying - thanks for the robust fix! It works for me! |
Cool, thanks for retesting it! ✌️ |
If it's always
Not sure how to reproduce this, since it only happened once. I was just playing around with the settings (threshold) and running the base against different browser. If I have a reproduce path I'll let you know, but I think this issue is not resolved completely yet. |
I'm seeing this issue still, but only with It's odd, because if I run the test in a different browser, I get an image diff of I've got This is the same whether I run it directly on the mac or via docker. node: v18.18.0 Edit: It doesn't seem to be due to the diff size, it just seems to be more reproducible if the diff is larger. Even with the Edit 2: I've also noticed that the test run counter is combining the number of tests run. So even though there are only 7 tests, when it re-runs twice, it shows 14 tests were run. If it re-runs three times, it shows 21, etc. Edit 3: It seems to not have a limit, it's just run 14 times (of a spec with 7 tests) and then ended up with a "No tests found" message, but it's passed 53 tests and failed 33, which isn't even the full number of runs for 14 runs ((53+33) / 14 =~ 12). Edit 4: I forgot to mention here, but I did a bit more digging and I think my issue is not related to this. Either my understanding is wrong, or there is a bug in the name counter cache (#300). |
Describe the bug
Similar to meinaart/cypress-plugin-snapshots#217 (and the open pull that fixes it),
cy.state('runnable')._currentRetry
(as documented here) should be incorporated into the nameCacheCounter logic so that a retried test is not seen as a new test.To Reproduce
Steps to reproduce the behavior:
retries
enabled in the cypress config (docs)Expected behavior
Test should go red. Instead test goes red, then on retry it goes green because the plugin is generating an actual with
#1.png
in the name instead of#0.png
and therefore treating it as a new test.Screenshots

Please complete the following information:
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: