BUG: DF.plot.hist() fails with "ValueError: setting an array element with a sequence" when using numpy 1.24

[rbu]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

X = pd.DataFrame({
    "p": ["A", "B"],
    "x1": [1, 1],
    "x2": [np.nan, 100.0],
})

X.plot.hist(by="p")

Issue Description

Generating a histogram plot that contains NaN values raises a ValueError when using numpy 1.24.

Searching for the error message, #50342 popped up. I have confirmed that this is a different bug and that neither pandas 1.5.3 nor 2.0.0 which contain that commit fix this.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_76881/2078980972.py in <cell line: 7>()
      5 })
      6 
----> 7 X.plot.hist(by="p")

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_core.py in hist(self, by, bins, **kwargs)
   1349             >>> ax = df.plot.hist(column=["age"], by="gender", figsize=(10, 8))
   1350         """
-> 1351         return self(kind="hist", by=by, bins=bins, **kwargs)
   1352 
   1353     def kde(self, bw_method=None, ind=None, **kwargs) -> PlotAccessor:

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_core.py in __call__(self, *args, **kwargs)
    973                     data.columns = label_name
    974 
--> 975         return plot_backend.plot(data, kind=kind, **kwargs)
    976 
    977     __call__.__doc__ = __doc__

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_matplotlib/__init__.py in plot(data, kind, **kwargs)
     69             kwargs["ax"] = getattr(ax, "left_ax", ax)
     70     plot_obj = PLOT_CLASSES[kind](data, **kwargs)
---> 71     plot_obj.generate()
     72     plot_obj.draw()
     73     return plot_obj.result

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_matplotlib/core.py in generate(self)
    446         self._compute_plot_data()
    447         self._setup_subplots()
--> 448         self._make_plot()
    449         self._add_table()
    450         self._make_legend()

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_matplotlib/hist.py in _make_plot(self)
    163                 kwds["weights"] = weights
    164 
--> 165             y = reformat_hist_y_given_by(y, self.by)
    166 
    167             artists = self._plot(ax, y, column_num=i, stacking_id=stacking_id, **kwds)

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_matplotlib/groupby.py in reformat_hist_y_given_by(y, by)
    136     """
    137     if by is not None and len(y.shape) > 1:
--> 138         return np.array([remove_na_arraylike(col) for col in y.T]).T
    139     return remove_na_arraylike(y)

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

Expected Behavior

I would expect to see a plot, or at least an error message explaining what is wrong with the data.

With numpy 1.23.X and earlier, I was getting a plot and this warning originating from pandas code:

/home/mldev/mambaforge/envs/instinct/lib/python3.9/site-packages/pandas/plotting/_matplotlib/groupby.py:138: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return np.array([remove_na_arraylike(col) for col in y.T]).T

Installed Versions

INSTALLED VERSIONS

commit : 478d340
python : 3.9.13.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.0-1035-azure
Version : #42~20.04.1-Ubuntu SMP Wed Mar 1 19:17:41 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : C.UTF-8
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.0
numpy : 1.24.2
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 59.5.0
pip : 23.0.1
Cython : None
pytest : 7.2.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 7.33.0
pandas_datareader: None
bs4 : 4.12.0
bottleneck : None
brotli :
fastparquet : None
fsspec : 2023.3.0
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : 2.7.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
snappy : None
sqlalchemy : 1.4.46
tables : 3.7.0
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : 2.3.1
pyqt5 : None

[Julian048]

@rbu is there a reason you're using X.plot.hist instead of X.hist?

[Julian048]

To my understanding np.nan values should be allowed as they work when the 'by' parameter is not passed, but if anyone could elaborate whether or not they should be allowed when 'by' is passed I would appreciate that.

[Julian048]

In the Code it looks intentional that nan values are not accepted, so I think an error message should be added in that case to let people know that nan values are not possible with the parameter by != None. (Unless the functionality to accept nan values should be added)