Description
From Discord:
Running into some weirdness.
In flutter/engine#50583, I added another set of tests running Android API v28, expecting new goldens. I only saw a few (<5) and assumed it just matched existing digests and moved on.
Now on an entirely unrelated PR, https://flutter-engine-gold.skia.org/cl/github/50601, they are showing up.
This is pretty crucial because these are the only rendering tests for the Android embedder, if I can't rely on the goldens all of this effort might be wasted.
I'll file a separate issue for some possible race-y conditions that result in a screenshot never being taken, but the test should definitely fail if an expected screenshot is not created and uploaded. @gaaclarke had some similar work for the Impeller golden files, and ended up with a diff file approach that seems reasonable to copy or at least be inspired by.
The outcome of this issue would be:
- We're sure that every test is taking and uploading a screenshot
- A missing screenshot (or failed upload) fails the test suite
- In combination with Filter the ADB logcat messages to the running application/test suite #143458, logging makes it clear what did/did not happen