groupby(multi-index level) not working correctly on a multi-indexed DataArray or DataSet

[emmaai]

What happened?

run the code block below with 2022.6.0

midx = pd.MultiIndex.from_product([list("abc"), [0, 1]], names=("one", "two"))

mda = xr.DataArray(np.random.rand(6, 3), [("x", midx), ("y", range(3))])

mda.groupby("one").groups

output:

In [15]: mda.groupby("one").groups
Out[15]: 
{('a', 0): [0],
 ('a', 1): [1],
 ('b', 0): [2],
 ('b', 1): [3],
 ('c', 0): [4],
 ('c', 1): [5]}

What did you expect to happen?

as it was with 2022.3.0

In [6]: mda.groupby("one").groups
Out[6]: {'a': [0, 1], 'b': [2, 3], 'c': [4, 5]}

Minimal Complete Verifiable Example

import pandas as pd
import numpy as np
import xarray as XR

midx = pd.MultiIndex.from_product([list("abc"), [0, 1]], names=("one", "two"))

mda = xr.DataArray(np.random.rand(6, 3), [("x", midx), ("y", range(3))])

mda.groupby("one").groups

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

N/A

Anything else we need to know?

N/A

Environment

INSTALLED VERSIONS

commit: None
python: 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 5.11.0-1025-aws
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.0
libnetcdf: 4.7.4

xarray: 2022.6.0
pandas: 1.4.3
numpy: 1.22.4
scipy: 1.7.3
netCDF4: 1.5.8
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.5.1.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.2.10
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2022.04.1
distributed: 2022.4.1
matplotlib: 3.5.1
cartopy: 0.20.3
seaborn: 0.11.2
numbagg: None
fsspec: 2022.01.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 45.2.0
pip: 22.2
conda: None
pytest: 7.1.2
IPython: 7.31.0
sphinx: None

[dcherian]

@benbovy I tracked this down to

>>> mda.one.to_index()
# v2022.06.0
MultiIndex([('a', 0),
            ('a', 1),
            ('b', 0),
            ('b', 1),
            ('c', 0),
            ('c', 1)],
           names=['one', 'two'])

# v2022.03.0
Index(['a', 'a', 'b', 'b', 'c', 'c'], dtype='object', name='x')

We call to_index here in safe_cast_to_index:

xarray/xarray/core/utils.py

Lines 115 to 140 in f8fee90

	def safe_cast_to_index(array: Any) -> pd.Index:
	"""Given an array, safely cast it to a pandas.Index.
	If it is already a pandas.Index, return it unchanged.
	Unlike pandas.Index, if the array has dtype=object or dtype=timedelta64,
	this function will not attempt to do automatic type conversion but will
	always return an index with dtype=object.
	"""
	if isinstance(array, pd.Index):
	index = array
	elif hasattr(array, "to_index"):
	# xarray Variable
	index = array.to_index()
	elif hasattr(array, "to_pandas_index"):
	# xarray Index
	index = array.to_pandas_index()
	elif hasattr(array, "array") and isinstance(array.array, pd.Index):
	# xarray PandasIndexingAdapter
	index = array.array
	else:
	kwargs = {}
	if hasattr(array, "dtype") and array.dtype.kind == "O":
	kwargs["dtype"] = object
	index = pd.Index(np.asarray(array), **kwargs)
	return _maybe_cast_to_cftimeindex(index)

Not sure if the fix should be only in the GroupBy specifically or more generally in safe_cast_to_index

The GroupBy context is

xarray/xarray/core/groupby.py

Line 434 in f8fee90

group_as_index = safe_cast_to_index(group)

[FabianHofmann]

After trying to dig down further into the code, I saw that grouping over levels seems to be broken generally (up-to-date main branch at time of writing), i.e.

import pandas as pd
import numpy as np
import xarray as xr

midx = pd.MultiIndex.from_product([list("abc"), [0, 1]], names=("one", "two"))
mda = xr.DataArray(np.random.rand(6, 3), [("x", midx), ("y", range(3))])
mda.groupby("one").sum()

raises:

  File ".../xarray/xarray/core/_reductions.py", line 5055, in sum
    return self.reduce(

  File ".../xarray/xarray/core/groupby.py", line 1191, in reduce
    return self.map(reduce_array, shortcut=shortcut)

  File ".../xarray/xarray/core/groupby.py", line 1095, in map
    return self._combine(applied, shortcut=shortcut)

  File ".../xarray/xarray/core/groupby.py", line 1127, in _combine
    index, index_vars = create_default_index_implicit(coord)

  File ".../xarray/xarray/core/indexes.py", line 974, in create_default_index_implicit
    index = PandasMultiIndex(array, name)

  File ".../xarray/xarray/core/indexes.py", line 552, in __init__
    raise ValueError(

ValueError: conflicting multi-index level name 'one' with dimension 'one'

in the function create_default_index_implicit. I am still a bit puzzled how to approach this. Any idea @benbovy?

[benbovy]

Thanks @emmaai for the issue report and thanks @dcherian and @FabianHofmann for tracking it down.

There is a lot of complexity related to pandas.MultiIndex special cases and it's been difficult to avoid new issues arising during the index refactor.

create_default_index_implicit has some hacks to create xarray objects directly from pandas.MultiIndex instances (e.g., xr.Dataset(coords={"x": pd_midx})) or even from xarray objects wrapping multi-indexes. The error raised here suggests that the issue should fixed before this call... Probably in safe_cast_to_index indeed.

We should probably avoid using .to_index() internally, or should we even deprecate it? The fact that mda.one.to_index() (in v2022.3.0) doesn't return the same result than mda.indexes["one"] adds more confusion than it adds value. Actually, in the long-term I'd be for deprecating all pandas.MultiIndex special cases in Xarray.

[benbovy]

From #7282 it looks like we need to convert the multi-index level to a single index when casting the group to an index. And from #7105 we can fix that in safe_cast_to_index() (sometimes the full multi-index is expected) so we probably need a special case in groupby.

[benbovy]

we can fix that in safe_cast_to_index()

...we cannot fix that in safe_cast_to_index() (or we can add a parameter to specify the desired result).

[mschrimpf]

Is there hope for groupby working on multi-indexed DataArrays again in the future? We -- and from the issue history it looks like others too -- are currently pinning xarray<2022.6 even though we would love to use newer versions.

[dcherian]

I think we could special-case extracting a multiindex level here:

xarray/xarray/core/groupby.py

Line 469 in d4db166

group = obj[group]

group at that stage should have values

['a', 'a', 'b', 'b', 'c', 'c']

@mschrimpf Can you try that and send in a PR?

[benbovy]

A special-case sounds reasonable to me as well as a temporary fix before looking into if/how we can refactor groupby so that it works with multiple kinds of built-in and/or custom indexes.

[emmaai]

I solved it temporarily by reset_index to groupby and set_xindex after, if anyone is looking.

[carynbear]

This is happening again :(

[max-sixty]

@carynbear please could you post a reproducible example? A new bug report would be ideal...