BUG: groupby.groups with NA categories fails

[rhshadrach]

• closes BUG: DataFrameGroupBy.groups fails when Categorical indexer contains NaNs and dropna=False #61356 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

There is a slight code duplication here, but we don't need to rely on Cateorical's codes because we can just directly use groupby's. We also can't use groupby to implement Index.groupby because the former only works in the case where the values are exhaustive.

[tehunter]

When both arguments are False, should NaN come after non-observed groups? That seems more intuitive to me, especially for an ordered categorical

[rhshadrach]

No - if you do an operation like sum the order here matches the order in that result.

[tehunter]

This is what I'm getting on both main and 2.2.3.

>>> df = DataFrame(
...         {"cat": Categorical(["a", np.nan, "a"], categories=list("adb"))},
...         index=list("xyz"),
...     )
>>> df["val"] = [1, 2, 3]
>>> g = df.groupby("cat", observed=False, dropna=False)
>>> g.sum()
     val
cat
a      4
d      0
b      0
NaN    2

[tehunter]

Ah, tm.assert_dict_equal appears to be order-invariant, so it doesn't matter for the test.

[rhshadrach]

Ah, I see now. I was correct in that the order was the same, but I failed to notice that the test added the groups in the incorrect order. I do wonder if assert_dict_equal should default to checking the order (perhaps with an argument to ignore order).

[mroeschke]

Thanks @rhshadrach