Failing main branch — test_save_mfdataset_compute_false_roundtrip

[max-sixty]

We had the main branch passing for a while, but unfortunately another test failure. Now in our new Linux py38-backend-api-v2 test case, intest_save_mfdataset_compute_false_roundtrip

link

self = <xarray.tests.test_backends.TestDask object at 0x7f821a0d6190>

    def test_save_mfdataset_compute_false_roundtrip(self):
        from dask.delayed import Delayed
    
        original = Dataset({"foo": ("x", np.random.randn(10))}).chunk()
        datasets = [original.isel(x=slice(5)), original.isel(x=slice(5, 10))]
        with create_tmp_file(allow_cleanup_failure=ON_WINDOWS) as tmp1:
            with create_tmp_file(allow_cleanup_failure=ON_WINDOWS) as tmp2:
                delayed_obj = save_mfdataset(
                    datasets, [tmp1, tmp2], engine=self.engine, compute=False
                )
                assert isinstance(delayed_obj, Delayed)
                delayed_obj.compute()
                with open_mfdataset(
                    [tmp1, tmp2], combine="nested", concat_dim="x"
                ) as actual:
>                   assert_identical(actual, original)
E                   AssertionError: Left and right Dataset objects are not identical
E                   
E                   
E                   Differing data variables:
E                   L   foo      (x) float64 dask.array<chunksize=(5,), meta=np.ndarray>
E                   R   foo      (x) float64 dask.array<chunksize=(10,), meta=np.ndarray>

/home/vsts/work/1/s/xarray/tests/test_backends.py:3274: AssertionError

AssertionError: Left and right Dataset objects are not identical

Differing data variables:
L   foo      (x) float64 dask.array<chunksize=(5,), meta=np.ndarray>
R   foo      (x) float64 dask.array<chunksize=(10,), meta=np.ndarray>

@aurghs & @alexamici — are you familiar with this? Thanks in advance

[keewis]

this seems to be flaky: #4532 doesn't touch backend related files, but while the CI run of 788e820 did fail, c0ae7c8 did not.

[aurghs]

I've tried to replicate the error, but I couldn't.

[keewis]

it also seems to be unrelated to the backend refactor: in #4530, the py38 CI failed

[keewis]

this is starting to pop up more frequently in PRs so it would be good to fix this. However, without a good way to reproduce this I guess it is really hard to debug (I tried

for ((i=0; i < 10; ++i)); do
    if ! python -m pytest -n 2 xarray/tests/test_backends.py; then
        break
    fi
done

but couldn't reproduce – maybe increasing the number of attempts or using a faster machine might work?)

[max-sixty]

Tests have been really flaky the past few days. I searched for the test that had just failed and — lo and behold! — an issue of mine from three years ago on a very similar test! 😁 (Edit: though not the same test — I'm not claiming it's been failing for years!)

If we can't easily fix it, would it be better to add an xfail? At the moment we're ignoring it anyway. CI red vs. green has been mostly noise for me since contributing again in the past week. (Looking at the full results is ofc useful, but the pass/fail is mostly meaningless).

Unless others have a different experience?

[headtr1ck]

If we can't easily fix it, would it be better to add an xfail? At the moment we're ignoring it anyway. CI red vs. green has been mostly noise for me since contributing again in the past week. (Looking at the full results is ofc useful, but the pass/fail is mostly meaningless).

Unless others have a different experience

Since it occasionally passes, x fail will not work. Maybe we have to skip it.

Edit: this seems to be wrong, sry

[max-sixty]

Since it occasionally passes, x fail will not work. Maybe we have to skip it.

Skipping is fine — though are xpass-es treated as failures? That's possible with pytest, but I don't see it in the configs...

[kmuehlbauer]

Only @pytest.mark.xfail(strict=True) will fail the test suite on XPASS (default is False) https://docs.pytest.org/en/stable/how-to/skipping.html#strict-parameter

[keewis]

... and we already have a bunch of xpasses that probably need to be investigated

[keewis]

actually, it is probably best to skip that test since it occasionally segfaults

[headtr1ck]

It seems that

xarray/tests/test_backends.py::test_open_mfdataset_manyfiles[netcdf4-20-True-None-5

Also fails occasionally