Feature: snapshot mismatch reporting callback

[coreyfarrell]

I'd like to propose that the options object of t.snapshot support a callback function to override mismatch error handling. My proposal would allow something like the following (or something similar):

t.snapshot(await ele.takeScreenshot(), {
  mismatchFn(expected, actual) {
    const fileExpected = 'snapshot-expected.png';
    const fileActual = 'snapshot-actual.png';
    const printErr = (err) => { if (err) console.error(err); };
    fs.writeFile(fileExpected, expected, {encoding:'base64'}, printErr);
    fs.writeFile(fileActual, actual, {encoding:'base64'}, printErr);
    // t.is replaced the assertion that t.snapshot isn't producing.
    t.is(fileExpected, fileActual, 'Snapshot images do not match');
  }
});

await ele.takeScreenshot() produces a base64 encoded png of the space within a specific browser element. t.snapshot allows me to check that rendering compared to the previous run. This proposal would create an easy way to replace the default call to fail() for mismatched snapshot values with one that processes and records the before/after information, replaces the AssertionError with a more useful message. The current AssertError I get for a difference between two images is just two large base64 dumps, my example would show the before/after filenames.

A couple alternatives:

• Allow a callback function that saves the data then returns [fileExpected, fileActual] so t.snapshot could still run the actual assertion but using the returned values for expected/actual?
• A less generic option to record data to before/after files and print filenames in the assertion instead of raw data? Would still need a way to control how actual encoding/rendering of the before/after files.

I'm sure of the feature I want, unsure the best API to expose it. I know I could probably reimplement t.snapshot to my own test helper but I suspect this could be generally useful. I can provide a PR once I get some feedback on the idea.

[novemberborn]

On the one hand, I'd like to have plugins for t.snapshot() and even t.deepEqual(), which could generate their own diffs.

#1692 would deliver an API you could use to build your mismatchFn implementation. What do you think of that proposal?

(I'm closing this issue for housekeeping purposes, but let's keep the conversation going.)

[coreyfarrell]

Thanks for replying. I'm not ignoring you it's just taking me a bit to digest 1692. I'm not sure yet how that would allow me to accomplish my goals which is to manipulate the reporting functionality of t.snapshot (both the console output and the test.js.md report).

That said this idea of what I need has had some time on the back-burner so I'm starting to think of it a bit differently. I'm thinking for the test.js.md output I could probably reformat the base64 png images so they contain the proper markdown:

t.snapshot(`![](data:image/png;base64,${imageBase64}]`);

If I need to look at two versions I can just copy the md to another file before updating it, render both and view in the browser. But this still means that upon failure I'll get two giant base64 dump's to the console output.

[novemberborn]

Yea the ideal would be something like:

import Image from 'ava-image-snapshot'

test('image is the same', t => {
  t.snapshot(Image.from(buffer))
})

But that would require the fictional ava-image-snapshot package to actually integrate with AVA and the underlying systems, which is going to take a lot more work under the hood to make possible and keep compatible.

The idea with #1692 is that you could do something like:

test('image is the same', async t => {
  const result = await t.try(t => {
    t.snapshot(imageBase64)
  })

  if (result.passed) {
    result.commit()
  } else {
    t.fail('image changed')
  }
})

This would be better with a helper function, so you'd end up with test('image is the same', checkImage, imageBase64).