Result of .snapshot() assertions in --update-snapshots mode is inconsistent

[ninevra]

What you're trying to do

Run AVA in --update-snapshots mode.

What happened

Whether t.snapshot() assertions pass or fail depends on the state of other tests in the same file, and can differ between files.

Specifically, t.snapshot() assertions always pass in --update-snapshots mode, unless any test in their file is test.skip()ed, in which case they pass or fail as they would without --update-snapshots.

What you expected to happen

I'd expect that, whatever the behavior of t.snapshot() assertions is here, it would be consistent, or at least not change as a result of changes to unrelated tests.

This isn't a bug, precisely; the documentation doesn't specify whether snapshot assertions will or won't pass in these situations. It's just odd.

MRE.

AVA configuration: none.

AVA version: 3.14.0.

[novemberborn]

We don't want to update snapshots if the resulting file may be incomplete due to tests or assertions being skipped. It's a little too easy to accidentally drop snapshots that way.

We may have implemented that as an assertion error, whereas instead we should have a way to "fail" the test file itself.

[novemberborn]

@ninevra was this addressed in your refactor or does this remain outstanding?

[ninevra]

At the time I didn't think snapshots v3 addressed this, but looking at it again, I guess it did. The refactor allows using skip()s freely while updating snapshots, which means snapshot assertions should now always pass in --update-snapshots mode.

I've just quickly checked this both against the MRE and against the code and everything looks fine.

Good catch!