Control output directory of snapshot tests

[doozMen]

Is it possible to control the output directory of the snapshot that is used for compare? I do not mean the reference image but the value to compare. The use case of this would be to have the snapshots downloadable as an artifact in a ci.

For example when using Swift Testing on bitrise the output generates an xctestresult but that one has images attached for XCTests with snapshots, but it has not for Swift Testing.

Issue recorded: Snapshot does not match reference.

ksdiff /Users/vagrant/git/Tests/SportPreviewTests/__Snapshots__/LineUpTests/injuredPlayer.1.png /Users/vagrant/Library/Developer/CoreSimulator/Devices/7E7282A4-118C-4B85-987B-D0A932FB88C3/data/tmp/LineUpTests/injuredPlayer.1.png

The percentage of pixels that match 0.9999879 is less than required 1.0
The lowest perceptual color precision 0.916875 is less than required 0.98

As compared to XCTests:

This might also be an issue of Bitrise issue but for now I would like to work around it by controlling where the test value is recorded /Users/vagrant/Library/Developer/CoreSimulator/Devices/7E7282A4-118C-4B85-987B-D0A932FB88C3/data/tmp/LineUpTests/injuredPlayer.1.png this is not really an easy location to put into an artifact. Is there a way to control this?

Thanks for the info!

[mbrandonw]

For example when using Swift Testing on bitrise the output generates an xctestresult but that one has images attached for XCTests with snapshots, but it has not for Swift Testing.

This is just because Swift Testing does not support attachments. There is a proposal for this now. If you want attachments, you can stick with XCTest in the meantime.

Since this isn't an issue with the library I am going to convert it to a discussion. Please feel free to continue the conversation over there!

[doozMen]

@mbrandonw thanks for the quick response. I should have put it in the discussion as this indeed is not an issue of the library.

I however have to get it working now and If you have time would like advice on the approach. I could potentially make a pull request of it if it meets your standards.

So my question. I have looked at the code and found that the AssertSnapshot.swift file has SNAPSHOT_ARTIFACTS as environment value that I could use to have the files output to a directory that I can manage.

      let data = try Data(contentsOf: snapshotFileUrl)
      let reference = snapshotting.diffing.fromData(data)

      #if os(iOS) || os(tvOS)
        // If the image generation fails for the diffable part and the reference was empty, use the reference
        if let localDiff = diffable as? UIImage,
          let refImage = reference as? UIImage,
          localDiff.size == .zero && refImage.size == .zero
        {
          diffable = reference
        }
      #endif

    // When running swift testing this exists as snapshotting.diffing.diff(reference, diffable) returns without attachements
      guard let (failure, attachments) = snapshotting.diffing.diff(reference, diffable) else {
        return nil
      }

      let artifactsUrl = URL(
        fileURLWithPath: ProcessInfo.processInfo.environment["SNAPSHOT_ARTIFACTS"]
          ?? NSTemporaryDirectory(), isDirectory: true
      )

So as there is a reference image already would it not be possible to just store that image to a temporary file that I can set as an artifact in CI?

[Archdoog]

While there's definitely a Swift Testing discussion here. We're also seeing this issue on projects using only XCTests that have been running for some time with no problem. It's a recent thing, where suddenly after something updated we're frequently seeing the simulator filesystem get used to save a snapshot, even after the library successfully writes the snapshot to the original __Snapshots__ directory beside the file. We noticed it first here, but now it's happening both on CI and locally and multiple projects.

/Users/jacob/Git/ferrostar/apple/Tests/TestSupport/AssertSnapshot.swift:26: error: -[FerrostarMapLibreUITests.LandscapeOverlayViewTests testDefault] : failed - Snapshot does not match reference.

@−
"file:///Users/jacob/Git/ferrostar/apple/Tests/TestSupport/__Snapshots__/AssertSnapshot/assertView-named-record-colorScheme-navigationUITheme-navigationFormatterCollection-frame-content-file-testName-line.1.png"
@+
"file:///Users/jacob/Library/Developer/CoreSimulator/Devices/0515888D-7AEF-4AC7-BAE4-D8CFB7E06FED/data/tmp/AssertSnapshot/assertView-named-record-colorScheme-navigationUITheme-navigationFormatterCollection-frame-content-file-testName-line.1.png"

I've tried all sorts of things like revising our calls to the assertSnapshot func, setting the environment variable SNAPSHOT_ARTIFACTS, etc. In each case it will sometimes correctly generate the snapshot in the expected folder (e.g. in ./__SNAPSHOTS___), other times on the simulator in tmp.

Happy to move this back to the GitHub issue if it's re-opened as I know this isn't relevant for the Swift Testing discussion.