Tests fail when installing h5netcdf without netcdf4

[caenrigen]

What happened:

Created a new conda env for xarray as described in the contribution guide.
Then I installed the h5netcdf and run all the tests (without making any changed in the xarray repo).
Tests failed due to missing dependencies apparently.

What you expected to happen:

Tests should not fail if a package is not a requirement and the backends can operate independently.

Minimal Complete Verifiable Example:

cd xarray
pip install -e .
pip install h5netcdf pytest-xdist
py.test -n 2 .

======================================================================== FAILURES =========================================================================
__________________________________________________________ TestH5NetCDFAlreadyOpen.test_deepcopy __________________________________________________________
[gw0] darwin -- Python 3.8.8 /usr/local/anaconda3/envs/xarray/bin/python3.8

self = <xarray.tests.test_backends.TestH5NetCDFAlreadyOpen object at 0x11ce5d5e0>

    def test_deepcopy(self):
        import h5netcdf
    
        with create_tmp_file() as tmp_file:
>           with nc4.Dataset(tmp_file, mode="w") as nc:
E           NameError: name 'nc4' is not defined

/Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/tests/test_backends.py:2717: NameError
_____________________________________________________ TestH5NetCDFAlreadyOpen.test_open_dataset_group _____________________________________________________
[gw1] darwin -- Python 3.8.8 /usr/local/anaconda3/envs/xarray/bin/python3.8

self = <xarray.tests.test_backends.TestH5NetCDFAlreadyOpen object at 0x11cfe6bb0>

    def test_open_dataset_group(self):
        import h5netcdf
    
        with create_tmp_file() as tmp_file:
>           with nc4.Dataset(tmp_file, mode="w") as nc:
E           NameError: name 'nc4' is not defined

/Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/tests/test_backends.py:2690: NameError
_____________________________________________________________ test_load_single_value_h5netcdf _____________________________________________________________
[gw0] darwin -- Python 3.8.8 /usr/local/anaconda3/envs/xarray/bin/python3.8

    def _get_default_engine_netcdf():
        try:
>           import netCDF4  # noqa: F401
E           ModuleNotFoundError: No module named 'netCDF4'

/Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/backends/api.py:105: ModuleNotFoundError

During handling of the above exception, another exception occurred:

    def _get_default_engine_netcdf():
        try:
            import netCDF4  # noqa: F401
    
            engine = "netcdf4"
        except ImportError:  # pragma: no cover
            try:
>               import scipy.io.netcdf  # noqa: F401
E               ModuleNotFoundError: No module named 'scipy'

/Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/backends/api.py:110: ModuleNotFoundError

During handling of the above exception, another exception occurred:

tmp_path = PosixPath('/private/var/folders/9k/k5388g_s44b_97319sslv9c40000gn/T/pytest-of-Victor/pytest-3/popen-gw0/test_load_single_value_h5netcd0')

    @requires_h5netcdf
    def test_load_single_value_h5netcdf(tmp_path):
        """Test that numeric single-element vector attributes are handled fine.
    
        At present (h5netcdf v0.8.1), the h5netcdf exposes single-valued numeric variable
        attributes as arrays of length 1, as oppesed to scalars for the NetCDF4
        backend.  This was leading to a ValueError upon loading a single value from
        a file, see #4471.  Test that loading causes no failure.
        """
        ds = xr.Dataset(
            {
                "test": xr.DataArray(
                    np.array([0]), dims=("x",), attrs={"scale_factor": 1, "add_offset": 0}
                )
            }
        )
>       ds.to_netcdf(tmp_path / "test.nc")

/Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/tests/test_backends.py:4993: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/core/dataset.py:1689: in to_netcdf
    return to_netcdf(
/Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/backends/api.py:1052: in to_netcdf
    engine = _get_default_engine(path_or_file)
/Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/backends/api.py:145: in _get_default_engine
    engine = _get_default_engine_netcdf()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def _get_default_engine_netcdf():
        try:
            import netCDF4  # noqa: F401
    
            engine = "netcdf4"
        except ImportError:  # pragma: no cover
            try:
                import scipy.io.netcdf  # noqa: F401
    
                engine = "scipy"
            except ImportError:
>               raise ValueError(
                    "cannot read or write netCDF files without "
                    "netCDF4-python or scipy installed"
                )
E               ValueError: cannot read or write netCDF files without netCDF4-python or scipy installed

/Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/backends/api.py:114: ValueError
==================================================================== warnings summary =====================================================================
xarray/tests/test_accessor_str.py: 18 warnings
  /Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/core/duck_array_ops.py:248: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
    flag_array = (arr1 == arr2) | (isnull(arr1) & isnull(arr2))

xarray/tests/test_dataset.py::TestDataset::test_resample_loffset
  /Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/tests/test_dataset.py:3904: FutureWarning: 'loffset' in .resample() and in Grouper() is deprecated.
  
  >>> df.resample(freq="3s", loffset="8H")
  
  becomes:
  
  >>> from pandas.tseries.frequencies import to_offset
  >>> df = df.resample(freq="3s").mean()
  >>> df.index = df.index.to_timestamp() + to_offset("8H")
  
    ds.bar.to_series().resample("24H", loffset="-12H").mean()

xarray/tests/test_dataarray.py::TestDataArray::test_resample
  /Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/tests/test_dataarray.py:2994: FutureWarning: 'loffset' in .resample() and in Grouper() is deprecated.
  
  >>> df.resample(freq="3s", loffset="8H")
  
  becomes:
  
  >>> from pandas.tseries.frequencies import to_offset
  >>> df = df.resample(freq="3s").mean()
  >>> df.index = df.index.to_timestamp() + to_offset("8H")
  
    expected = DataArray(array.to_series().resample("24H", loffset="-12H").mean())

xarray/tests/test_dataarray.py::TestDataArray::test_polyfit[True-False]
xarray/tests/test_dataarray.py::TestDataArray::test_polyfit[False-False]
  /Users/Victor/Documents/ProjectsDev/Qblox/xarray/xarray/core/dataset.py:6396: RuntimeWarning: overflow encountered in multiply
    scale = np.sqrt((lhs * lhs).sum(axis=0))

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================= short test summary info =================================================================
FAILED xarray/tests/test_backends.py::TestH5NetCDFAlreadyOpen::test_deepcopy - NameError: name 'nc4' is not defined
FAILED xarray/tests/test_backends.py::TestH5NetCDFAlreadyOpen::test_open_dataset_group - NameError: name 'nc4' is not defined
FAILED xarray/tests/test_backends.py::test_load_single_value_h5netcdf - ValueError: cannot read or write netCDF files without netCDF4-python or scipy in...
============================== 3 failed, 3474 passed, 4472 skipped, 65 xfailed, 17 xpassed, 22 warnings in 66.86s (0:01:06) ===============================

Anything else we need to know?:

Tests still fail when installing scipy only.

They only pass after installing both netcdf4 and scipy.

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: 48378c4
python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:12:38)
[Clang 11.0.1 ]
python-bits: 64
OS: Darwin
OS-release: 18.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.12.0
libnetcdf: None

xarray: 0.17.1.dev3+g48378c4b
pandas: 1.2.2
numpy: 1.20.1
scipy: None
netCDF4: None
pydap: None
h5netcdf: 0.10.0
h5py: 3.1.0
Nio: None
zarr: None
cftime: 1.4.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 49.6.0.post20210108
pip: 21.0.1
conda: None
pytest: 6.2.2
IPython: None
sphinx: None

[shoyer]

Thanks for the heads up. These tests should be marked as also requiring netCDF4.

[max-sixty]

@caenrigen thanks for the issue. Would you be up for fixing this with a PR?

[caenrigen]

@caenrigen thanks for the issue. Would you be up for fixing this with a PR?

Unfortunately out of bandwidth currently, this came out when addressing #4981 so I reported it as suggested.

For anyone facing the issue, installing netcdf4 solves the issue as a temporary fix

[feefladder]

take