Test fails if simulator is changed

[tikekar]

Describe the bug
I had iphone 13 max simulator selected in my xcode project on which I ran the snapshot test to record. After that I changed the simulator and re ran the test. This time it fails with following error -
Newly-taken snapshot@(375.0, 812.0) does not match reference@(562.5, 1218.0).

But I have selected iphoneX in the image. Here is my sample testcase. So how to make this agnostic to simulator type? As I want to run this test on CI/CD, I would like the test to pass if a different simulator is chosen. Or is there a way to provide a constant simulator type to the test so that it always runs on same type.
let result = verifySnapshot(
matching: viewController,
as: .image(on: .iPhoneX), // Selected iphoneX
named: "MyPage",
record: false,
timeout: 20,
testName: "MyTest"
)

To Reproduce
Zip up a project that reproduces the behavior and attach it by dragging it here.

// And/or enter code that reproduces the behavior here.
let result = verifySnapshot(
            matching: viewController,
            as: .image(on: .iPhoneX), // Selected iphoneX
            named: "MyPage",
            record: false,
            timeout: 20,
            testName: "MyTest"
        )

Expected behavior
Test should pass even if the simulator it runs on is changed if the image type is provided.

Screenshots

hots to help explain your problem.

Environment

• swift-snapshot-testing version [e.g. 1.9.0]
• Xcode [e.g. 13.2]
• Swift [e.g. 5.5]
• OS: [e.g. iOS 15]

Additional context
Add any more context about the problem here.

[tikekar]

I have a few questions -

• If my image takes iphoneX and I if the test runs on iphone 13 max simulator, how does it work? Shouldn't the simulator type and image device match?
• As a follow up if I provide multiple device configs to be provided to the image, will they all run on same chosen simulator?

[gohanlon]

Unfortunately, the only way to capture a true "iPhone X" rendering is to run your test suite with that device simulator. The as: .image(on: .iPhoneX) sets up the dimensions and safe area of the view for which the captured image is taken, but there's a lot more to the rendering pipeline that can't be controlled.

Note the warning from the README:

⚠️ Warning: Snapshots must be compared using a simulator with the same OS, device gamut, and scale as the simulator that originally took the reference to avoid discrepancies between images.

Which, I think could be made stronger (see #592):

Other things that seem to cause differences are the architecture of the Mac running the simulator causing subtle "sub" pixel differences, and optical font sizing adjustments that either don't respect the displayScale UI trait or deliberately ignore it because it's truly driven by the device display's points per inch (a product of the display's scale and pixels per inch).

Perhaps it'd make sense to even caution directly against what you're trying to do (which I also tried!), since it's definitely a bit misleading. I think the authors originally intended for the library to be usable in this way, but ended up not finding a way to externally control enough of the rendering.

I've adopted a strategy of simply running many simulators instead of trying to make one simulator render like another. The simulators can be run in parallel, which isn't as fast as it could be (see #591), but otherwise works really well. This approach should be fairly robust to Apple adding new devices and other rendering changes that Apple makes over time.

If you think you'd like to try and replicate my approach, I put all the pieces together in this Gist: https://gist.github.com/gohanlon/d40e8205117d68e66732b8e6e85bd6cb

[gohanlon]

I just noticed your questions, which I think I mostly answered above, but I'll answer directly:

If my image takes iphoneX and I if the test runs on iphone 13 max simulator, how does it work? Shouldn't the simulator type and image device match?

Yes. If you need accurate results, the actual running simulator and the ViewImageConfig (e.g. ViewImageConfig.iPhoneX) need to match.

As a follow up if I provide multiple device configs to be provided to the image, will they all run on same chosen simulator?

Yes, all your images will be captured using the selected run destination (simulator), which will not produce accurate results.

[tikekar]

Hi Galen Thanks for replying. I have one more question. I am exploring git lfs to store these images. Will running the snapshot test on CICD pipeline work as it is? Currently it is failing for me. So wanted to check if there is anything else required to fetch images from the lfs storage for comparison. thanks Gauri

…

On Fri, Jul 8, 2022 at 12:03 PM Galen O’Hanlon ***@***.***> wrote: I just noticed your questions, which I think I mostly answered above, but I'll answer directly: If my image takes iphoneX and I if the test runs on iphone 13 max simulator, how does it work? Shouldn't the simulator type and image device match? Yes. If you need accurate results, the actual running simulator and the ViewImageConfig (e.g. ViewImageConfig.iPhoneX) need to match. As a follow up if I provide multiple device configs to be provided to the image, will they all run on same chosen simulator? Yes, all your images will be captured using the selected run destination (simulator), which will not produce accurate results. — Reply to this email directly, view it on GitHub <#614 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACA4RTLSV3CN5PG5TMIZ4ADVTB3OHANCNFSM527GNONA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[gohanlon]

I'm not running snapshot testing on CI, but it can certainly be done, including with git-lfs. How's it failing, exactly? Are you using Github Actions, or something else?

Keep in mind that, in addition to the exact device Simulator and OS that's used, the system architecture on which the Simulator is run will also render subtly different (i.e. Intel vs. Apple Silicon). #571 introduces a way to be resilient to these differences. Personally, I only run snapshot tests on local Apple Silicon dev machines, and plan to adopt CI when/if Github offers hosted runners on Apple Silicon.

[tikekar]

Thanks for the clarification. We use Jenkins so I think we will need to add some extra steps for enabling it to be able to fetch from lfs right? It doesn’t show somehow why it failed. But my guess is that it is not able to find the reference image . I am going to explore Jenkins on how to support lfs from there. Thanks Gauri

…

On Mon, Aug 22, 2022 at 6:11 PM Galen O’Hanlon ***@***.***> wrote: I'm not running snapshot testing on CI, but it can certainly be done, including with git-lfs. How's it failing, exactly? Are you using Github Actions, or something else? Keep in mind that, in addition to the exact device Simulator and OS that's used, the system architecture on which the Simulator is run will also render subtly different (i.e. Intel vs. Apple Silicon). #571 <#571> introduces a way to be resilient to these differences. Personally, I only run snapshot tests on local Apple Silicon dev machines, and plan to adopt CI when/if Github offers hosted runners on Apple Silicon. — Reply to this email directly, view it on GitHub <#614 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACA4RTNOQJG562ASB6WL4YDV2QQLDANCNFSM527GNONA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[gohanlon]

We use Jenkins so I think we will need to add some extra steps for enabling it to be able to fetch from lfs right?

Yes. You may just need a git lfs pull command before you run your tests. Good luck!

[tikekar]

Thanks! Yes we tried that and it worked.

…

On Mon, Aug 22, 2022 at 9:58 PM Galen O’Hanlon ***@***.***> wrote: We use Jenkins so I think we will need to add some extra steps for enabling it to be able to fetch from lfs right? Yes. You may just need a git lfs pull command before you run your tests. Good luck! — Reply to this email directly, view it on GitHub <#614 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACA4RTIVFL77K4MML5M2MJTV2RLAPANCNFSM527GNONA> . You are receiving this because you authored the thread.Message ID: ***@***.***>