v2.1.0.rst

What's new in 2.1.0 (Month XX, 2023)

These are the changes in pandas 2.1.0. See :ref:`release` for a full changelog including other versions of pandas.

{{ header }}

Enhancements

Copy-on-Write improvements

• :meth:`Series.transform` not respecting Copy-on-Write when func modifies :class:`Series` inplace (:issue:`53747`)
• Calling :meth:`Index.values` will now return a read-only NumPy array (:issue:`53704`)
• Setting a :class:`Series` into a :class:`DataFrame` now creates a lazy instead of a deep copy (:issue:`53142`)
• The :class:`DataFrame` constructor, when constructing a DataFrame from a dictionary of Index objects and specifying copy=False, will now use a lazy copy of those Index objects for the columns of the DataFrame (:issue:`52947`)

map(func, na_action="ignore") now works for all array types

When given a callable, :meth:`Series.map` applies the callable to all elements of the :class:`Series`. Similarly, :meth:`DataFrame.map` applies the callable to all elements of the :class:`DataFrame`, while :meth:`Index.map` applies the callable to all elements of the :class:`Index`.

Frequently, it is not desirable to apply the callable to nan-like values of the array and to avoid doing that, the map method could be called with na_action="ignore", i.e. ser.map(func, na_action="ignore"). However, na_action="ignore" was not implemented for many ExtensionArray and Index types and na_action="ignore" did not work correctly for any ExtensionArray subclass except the nullable numeric ones (i.e. with dtype :class:`Int64` etc.).

na_action="ignore" now works for all array types (:issue:`52219`, :issue:`51645`, :issue:`51809`, :issue:`51936`, :issue:`52033`; :issue:`52096`).

Previous behavior:

In [1]: ser = pd.Series(["a", "b", np.nan], dtype="category")
In [2]: ser.map(str.upper, na_action="ignore")
NotImplementedError
In [3]: df = pd.DataFrame(ser)
In [4]: df.applymap(str.upper, na_action="ignore")  # worked for DataFrame
     0
0    A
1    B
2  NaN
In [5]: idx = pd.Index(ser)
In [6]: idx.map(str.upper, na_action="ignore")
TypeError: CategoricalIndex.map() got an unexpected keyword argument 'na_action'

New behavior:

.. ipython:: python

    ser = pd.Series(["a", "b", np.nan], dtype="category")
    ser.map(str.upper, na_action="ignore")
    df = pd.DataFrame(ser)
    df.map(str.upper, na_action="ignore")
    idx = pd.Index(ser)
    idx.map(str.upper, na_action="ignore")

Notice also that in this version, :meth:`DataFrame.map` been added and :meth:`DataFrame.applymap` has been deprecated. :meth:`DataFrame.map` has the same functionality as :meth:`DataFrame.applymap`, but the new name better communicate that this is the :class:`DataFrame` version of :meth:`Series.map` (:issue:`52353`).

Also, note that :meth:`Categorical.map` implicitly has had its na_action set to "ignore" by default. This has been deprecated and will :meth:`Categorical.map` in the future change the default to na_action=None, like for all the other array types.

Other enhancements

• :meth:`Categorical.map` and :meth:`CategoricalIndex.map` now have a na_action parameter. :meth:`Categorical.map` implicitly had a default value of "ignore" for na_action. This has formally been deprecated and will be changed to None in the future. Also notice that :meth:`Series.map` has default na_action=None and calls to series with categorical data will now use na_action=None unless explicitly set otherwise (:issue:`44279`)
• :class:`api.extensions.ExtensionArray` now has a :meth:`~api.extensions.ExtensionArray.map` method (:issue:`51809`)
• :meth:`DataFrame.applymap` now uses the :meth:`~api.extensions.ExtensionArray.map` method of underlying :class:`api.extensions.ExtensionArray` instances (:issue:`52219`)
• :meth:`MultiIndex.sort_values` now supports na_position (:issue:`51612`)
• :meth:`MultiIndex.sortlevel` and :meth:`Index.sortlevel` gained a new keyword na_position (:issue:`51612`)
• :meth:`arrays.DatetimeArray.map`, :meth:`arrays.TimedeltaArray.map` and :meth:`arrays.PeriodArray.map` can now take a na_action argument (:issue:`51644`)
• :meth:`arrays.SparseArray.map` now supports na_action (:issue:`52096`).
• Add :meth:`diff()` and :meth:`round()` for :class:`Index` (:issue:`19708`)
• Add dtype of categories to repr information of :class:`CategoricalDtype` (:issue:`52179`)
• Added to the escape mode "latex-math" preserving without escaping all characters between "(" and ")" in formatter (:issue:`51903`)
• Adding engine_kwargs parameter to :meth:`DataFrame.read_excel` (:issue:`52214`)
• Classes that are useful for type-hinting have been added to the public API in the new submodule pandas.api.typing (:issue:`48577`)
• Implemented :attr:`Series.dt.is_month_start`, :attr:`Series.dt.is_month_end`, :attr:`Series.dt.is_year_start`, :attr:`Series.dt.is_year_end`, :attr:`Series.dt.is_quarter_start`, :attr:`Series.dt.is_quarter_end`, :attr:`Series.dt.is_days_in_month`, :attr:`Series.dt.unit`, :meth:`Series.dt.is_normalize`, :meth:`Series.dt.day_name`, :meth:`Series.dt.month_name`, :meth:`Series.dt.tz_convert` for :class:`ArrowDtype` with pyarrow.timestamp (:issue:`52388`, :issue:`51718`)
• Implemented __from_arrow__ on :class:`DatetimeTZDtype`. (:issue:`52201`)
• Implemented __pandas_priority__ to allow custom types to take precedence over :class:`DataFrame`, :class:`Series`, :class:`Index`, or :class:`ExtensionArray` for arithmetic operations, :ref:`see the developer guide <extending.pandas_priority>` (:issue:`48347`)
• Improve error message when having incompatible columns using :meth:`DataFrame.merge` (:issue:`51861`)
• Improve error message when setting :class:`DataFrame` with wrong number of columns through :meth:`DataFrame.isetitem` (:issue:`51701`)
• Improved error handling when using :meth:`DataFrame.to_json` with incompatible index and orient arguments (:issue:`52143`)
• Improved error message when creating a DataFrame with empty data (0 rows), no index and an incorrect number of columns. (:issue:`52084`)
• Let :meth:`DataFrame.to_feather` accept a non-default :class:`Index` and non-string column names (:issue:`51787`)
• Performance improvement in :func:`read_csv` (:issue:`52632`) with engine="c"
• :meth:`Categorical.from_codes` has gotten a validate parameter (:issue:`50975`)
• :meth:`DataFrame.stack` gained the sort keyword to dictate whether the resulting :class:`MultiIndex` levels are sorted (:issue:`15105`)
• :meth:`DataFrame.unstack` gained the sort keyword to dictate whether the resulting :class:`MultiIndex` levels are sorted (:issue:`15105`)
• :meth:`DataFrameGroupby.agg` and :meth:`DataFrameGroupby.transform` now support grouping by multiple keys when the index is not a :class:`MultiIndex` for engine="numba" (:issue:`53486`)
• :meth:`Series.explode` now supports pyarrow-backed list types (:issue:`53602`)
• :meth:`Series.str.join` now supports ArrowDtype(pa.string()) (:issue:`53646`)
• :meth:`SeriesGroupby.agg` and :meth:`DataFrameGroupby.agg` now support passing in multiple functions for engine="numba" (:issue:`53486`)
• :meth:`SeriesGroupby.transform` and :meth:`DataFrameGroupby.transform` now support passing in a string as the function for engine="numba" (:issue:`53579`)
• Added engine_kwargs parameter to :meth:`DataFrame.to_excel` (:issue:`53220`)
• Added a new parameter by_row to :meth:`Series.apply`. When set to False the supplied callables will always operate on the whole Series (:issue:`53400`).
• Groupby aggregations (such as :meth:`DataFrameGroupby.sum`) now can preserve the dtype of the input instead of casting to float64 (:issue:`44952`)
• Improved error message when :meth:`DataFrameGroupBy.agg` failed (:issue:`52930`)
• Many read/to_* functions, such as :meth:`DataFrame.to_pickle` and :func:`read_csv`, support forwarding compression arguments to lzma.LZMAFile (:issue:`52979`)
• Performance improvement in :func:`concat` with homogeneous np.float64 or np.float32 dtypes (:issue:`52685`)
• Performance improvement in :meth:`DataFrame.filter` when items is given (:issue:`52941`)

Notable bug fixes

These are bug fixes that might have notable behavior changes.

notable_bug_fix1

notable_bug_fix2

Backwards incompatible API changes

Increased minimum version for Python

pandas 2.1.0 supports Python 3.9 and higher.

Increased minimum versions for dependencies

Some minimum supported versions of dependencies were updated. If installed, we now require:

Package	Minimum Version	Required	Changed
numpy	1.21.6	X	X
mypy (dev)	1.2		X
beautifulsoup4	4.11.1		X
bottleneck	1.3.4		X
fastparquet	0.8.1		X
fsspec	2022.05.0		X
hypothesis	6.46.1		X
gcsfs	2022.05.0		X
jinja2	3.1.2		X
lxml	4.8.0		X
numba	0.55.2		X
numexpr	2.8.0		X
openpyxl	3.0.10		X
pandas-gbq	0.17.5		X
psycopg2	2.9.3		X
pyreadstat	1.1.5		X
pyqt5	5.15.6		X
pytables	3.7.0		X
pytest	7.3.2		X
python-snappy	0.6.1		X
pyxlsb	1.0.9		X
s3fs	2022.05.0		X
scipy	1.8.1		X
sqlalchemy	1.4.36		X
tabulate	0.8.10		X
xarray	2022.03.0		X
xlsxwriter	3.0.3		X
zstandard	0.17.0		X

For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

Package	Minimum Version	Changed
		X

See :ref:`install.dependencies` and :ref:`install.optional_dependencies` for more.

Other API changes

Deprecations

• Deprecated 'broadcast_axis' keyword in :meth:`Series.align` and :meth:`DataFrame.align`, upcast before calling align with left = DataFrame({col: left for col in right.columns}, index=right.index) (:issue:`51856`)
• Deprecated 'fill_method' and 'limit' keywords in :meth:`DataFrame.pct_change`, :meth:`Series.pct_change`, :meth:`DataFrameGroupBy.pct_change`, and :meth:`SeriesGroupBy.pct_change`, explicitly call ffill or bfill before calling pct_change instead (:issue:`53491`)
• Deprecated 'method', 'limit', and 'fill_axis' keywords in :meth:`DataFrame.align` and :meth:`Series.align`, explicitly call fillna on the alignment results instead (:issue:`51856`)
• Deprecated 'quantile' keyword in :meth:`Rolling.quantile` and :meth:`Expanding.quantile`, renamed as 'q' instead (:issue:`52550`)
• Deprecated :meth:`.DataFrameGroupBy.apply` and methods on the objects returned by :meth:`.DataFrameGroupBy.resample` operating on the grouping column(s); select the columns to operate on after groupby to either explicitly include or exclude the groupings and avoid the FutureWarning (:issue:`7155`)
• Deprecated :meth:`.Groupby.all` and :meth:`.GroupBy.any` with datetime64 or :class:`PeriodDtype` values, matching the :class:`Series` and :class:`DataFrame` deprecations (:issue:`34479`)
• Deprecated :meth:`Categorical.to_list`, use obj.tolist() instead (:issue:`51254`)
• Deprecated :meth:`DataFrame._data` and :meth:`Series._data`, use public APIs instead (:issue:`33333`)
• Deprecated :meth:`DataFrameGroupBy.dtypes`, check dtypes on the underlying object instead (:issue:`51045`)
• Deprecated axis=1 in :meth:`DataFrame.ewm`, :meth:`DataFrame.rolling`, :meth:`DataFrame.expanding`, transpose before calling the method instead (:issue:`51778`)
• Deprecated axis=1 in :meth:`DataFrame.groupby` and in :class:`Grouper` constructor, do frame.T.groupby(...) instead (:issue:`51203`)
• Deprecated accepting slices in :meth:`DataFrame.take`, call obj[slicer] or pass a sequence of integers instead (:issue:`51539`)
• Deprecated explicit support for subclassing :class:`Index` (:issue:`45289`)
• Deprecated making functions given to :meth:`Series.agg` attempt to operate on each element in the :class:`Series` and only operate on the whole :class:`Series` if the elementwise operations failed. In the future, functions given to :meth:`Series.agg` will always operate on the whole :class:`Series` only. To keep the current behavior, use :meth:`Series.transform` instead. (:issue:`53325`)
• Deprecated making the functions in a list of functions given to :meth:`DataFrame.agg` attempt to operate on each element in the :class:`DataFrame` and only operate on the columns of the :class:`DataFrame` if the elementwise operations failed. To keep the current behavior, use :meth:`DataFrame.transform` instead. (:issue:`53325`)
• Deprecated passing a :class:`DataFrame` to :meth:`DataFrame.from_records`, use :meth:`DataFrame.set_index` or :meth:`DataFrame.drop` instead (:issue:`51353`)
• Deprecated silently dropping unrecognized timezones when parsing strings to datetimes (:issue:`18702`)
• Deprecated the axis keyword in :meth:`DataFrame.ewm`, :meth:`Series.ewm`, :meth:`DataFrame.rolling`, :meth:`Series.rolling`, :meth:`DataFrame.expanding`, :meth:`Series.expanding` (:issue:`51778`)
• Deprecated the axis keyword in :meth:`DataFrame.resample`, :meth:`Series.resample` (:issue:`51778`)
• Deprecated the behavior of :func:`concat` with both len(keys) != len(objs), in a future version this will raise instead of truncating to the shorter of the two sequences (:issue:`43485`)
• Deprecated the default of observed=False in :meth:`DataFrame.groupby` and :meth:`Series.groupby`; this will default to True in a future version (:issue:`43999`)
• Deprecating pinning group.name to each group in :meth:`SeriesGroupBy.aggregate` aggregations; if your operation requires utilizing the groupby keys, iterate over the groupby object instead (:issue:`41090`)
• Deprecated the 'axis' keyword in :meth:`.GroupBy.idxmax`, :meth:`.GroupBy.idxmin`, :meth:`.GroupBy.fillna`, :meth:`.GroupBy.take`, :meth:`.GroupBy.skew`, :meth:`.GroupBy.rank`, :meth:`.GroupBy.cumprod`, :meth:`.GroupBy.cumsum`, :meth:`.GroupBy.cummax`, :meth:`.GroupBy.cummin`, :meth:`.GroupBy.pct_change`, :meth:`GroupBy.diff`, :meth:`.GroupBy.shift`, and :meth:`DataFrameGroupBy.corrwith`; for axis=1 operate on the underlying :class:`DataFrame` instead (:issue:`50405`, :issue:`51046`)
• Deprecated :class:`.DataFrameGroupBy` with as_index=False not including groupings in the result when they are not columns of the DataFrame (:issue:`49519`)
• Deprecated :func:`is_categorical_dtype`, use isinstance(obj.dtype, pd.CategoricalDtype) instead (:issue:`52527`)
• Deprecated :func:`is_datetime64tz_dtype`, check isinstance(dtype, pd.DatetimeTZDtype) instead (:issue:`52607`)
• Deprecated :func:`is_int64_dtype`, check dtype == np.dtype(np.int64) instead (:issue:`52564`)
• Deprecated :func:`is_interval_dtype`, check isinstance(dtype, pd.IntervalDtype) instead (:issue:`52607`)
• Deprecated :func:`is_period_dtype`, check isinstance(dtype, pd.PeriodDtype) instead (:issue:`52642`)
• Deprecated :func:`is_sparse`, check isinstance(dtype, pd.SparseDtype) instead (:issue:`52642`)
• Deprecated :meth:`.Styler.applymap_index`. Use the new :meth:`.Styler.map_index` method instead (:issue:`52708`)
• Deprecated :meth:`.Styler.applymap`. Use the new :meth:`.Styler.map` method instead (:issue:`52708`)
• Deprecated :meth:`DataFrame.applymap`. Use the new :meth:`DataFrame.map` method instead (:issue:`52353`)
• Deprecated :meth:`DataFrame.swapaxes` and :meth:`Series.swapaxes`, use :meth:`DataFrame.transpose` or :meth:`Series.transpose` instead (:issue:`51946`)
• Deprecated freq parameter in :class:`PeriodArray` constructor, pass dtype instead (:issue:`52462`)
• Deprecated allowing non-standard inputs in :func:`take`, pass either a numpy.ndarray, :class:`ExtensionArray`, :class:`Index`, or :class:`Series` (:issue:`52981`)
• Deprecated allowing non-standard sequences for :func:`isin`, :func:`value_counts`, :func:`unique`, :func:`factorize`, case to one of numpy.ndarray, :class:`Index`, :class:`ExtensionArray`, or :class:`Series` before calling (:issue:`52986`)
• Deprecated behavior of :class:`DataFrame` reductions sum, prod, std, var, sem with axis=None, in a future version this will operate over both axes returning a scalar instead of behaving like axis=0; note this also affects numpy functions e.g. np.sum(df) (:issue:`21597`)
• Deprecated behavior of :func:`concat` when :class:`DataFrame` has columns that are all-NA, in a future version these will not be discarded when determining the resulting dtype (:issue:`40893`)
• Deprecated behavior of :meth:`Series.dt.to_pydatetime`, in a future version this will return a :class:`Series` containing python datetime objects instead of an ndarray of datetimes; this matches the behavior of other :meth:`Series.dt` properties (:issue:`20306`)
• Deprecated logical operations (|, &, ^) between pandas objects and dtype-less sequences (e.g. list, tuple), wrap a sequence in a :class:`Series` or numpy array before operating instead (:issue:`51521`)
• Deprecated making :meth:`Series.apply` return a :class:`DataFrame` when the passed-in callable returns a :class:`Series` object. In the future this will return a :class:`Series` whose values are themselves :class:`Series`. This pattern was very slow and it's recommended to use alternative methods to archive the same goal (:issue:`52116`)
• Deprecated parameter convert_type in :meth:`Series.apply` (:issue:`52140`)
• Deprecated passing a dictionary to :meth:`.SeriesGroupBy.agg`; pass a list of aggregations instead (:issue:`50684`)
• Deprecated the "fastpath" keyword in :class:`Categorical` constructor, use :meth:`Categorical.from_codes` instead (:issue:`20110`)
• Deprecated the behavior of :func:`is_bool_dtype` returning True for object-dtype :class:`Index` of bool objects (:issue:`52680`)
• Deprecated the methods :meth:`Series.bool` and :meth:`DataFrame.bool` (:issue:`51749`)
• Deprecated unused "closed" and "normalize" keywords in the :class:`DatetimeIndex` constructor (:issue:`52628`)
• Deprecated unused "closed" keyword in the :class:`TimedeltaIndex` constructor (:issue:`52628`)
• Deprecated logical operation between two non boolean :class:`Series` with different indexes always coercing the result to bool dtype. In a future version, this will maintain the return type of the inputs. (:issue:`52500`, :issue:`52538`)
• Deprecated :func:`value_counts`, use pd.Series(obj).value_counts() instead (:issue:`47862`)
• Deprecated :meth:`Series.first` and :meth:`DataFrame.first` (please create a mask and filter using .loc instead) (:issue:`45908`)
• Deprecated :meth:`Series.interpolate` and :meth:`DataFrame.interpolate` for object-dtype (:issue:`53631`)
• Deprecated :meth:`Series.last` and :meth:`DataFrame.last` (please create a mask and filter using .loc instead) (:issue:`53692`)
• Deprecated allowing downcast keyword other than None, False, "infer", or a dict with these as values in :meth:`Series.fillna`, :meth:`DataFrame.fillna` (:issue:`40988`)
• Deprecated allowing arbitrary fill_value in :class:`SparseDtype`, in a future version the fill_value will need to be compatible with the dtype.subtype, either a scalar that can be held by that subtype or NaN for integer or bool subtypes (:issue:`23124`)
• Deprecated behavior of :func:`assert_series_equal` and :func:`assert_frame_equal` considering NA-like values (e.g. NaN vs None as equivalent) (:issue:`52081`)
• Deprecated constructing :class:`SparseArray` from scalar data, pass a sequence instead (:issue:`53039`)
• Deprecated falling back to filling when value is not specified in :meth:`DataFrame.replace` and :meth:`Series.replace` with non-dict-like to_replace (:issue:`33302`)
• Deprecated literal json input to :func:`read_json`. Wrap literal json string input in io.StringIO instead. (:issue:`53409`)
• Deprecated literal string/bytes input to :func:`read_html`. Wrap literal string/bytes input in io.StringIO / io.BytesIO instead. (:issue:`53767`)
• Deprecated option "mode.use_inf_as_na", convert inf entries to NaN before instead (:issue:`51684`)
• Deprecated parameter obj in :meth:`GroupBy.get_group` (:issue:`53545`)
• Deprecated positional indexing on :class:`Series` with :meth:`Series.__getitem__` and :meth:`Series.__setitem__`, in a future version ser[item] will always interpret item as a label, not a position (:issue:`50617`)
• Deprecated strings T, t, L and l denoting units in :func:`to_timedelta` (:issue:`52536`)
• Deprecated the "method" and "limit" keywords on :meth:`Series.fillna`, :meth:`DataFrame.fillna`, :meth:`SeriesGroupBy.fillna`, :meth:`DataFrameGroupBy.fillna`, and :meth:`Resampler.fillna`, use obj.bfill() or obj.ffill() instead (:issue:`53394`)
• Deprecated the method and limit keywords in :meth:`DataFrame.replace` and :meth:`Series.replace` (:issue:`33302`)
• Deprecated values "pad", "ffill", "bfill", "backfill" for :meth:`Series.interpolate` and :meth:`DataFrame.interpolate`, use obj.ffill() or obj.bfill() instead (:issue:`53581`)

Performance improvements

• Performance improvement in :func:`factorize` for object columns not containing strings (:issue:`51921`)
• Performance improvement in :func:`read_orc` when reading a remote URI file path. (:issue:`51609`)
• Performance improvement in :func:`read_parquet` and :meth:`DataFrame.to_parquet` when reading a remote file with engine="pyarrow" (:issue:`51609`)
• Performance improvement in :func:`read_parquet` on string columns when using use_nullable_dtypes=True (:issue:`47345`)
• Performance improvement in :meth:`DataFrame.clip` and :meth:`Series.clip` (:issue:`51472`)
• Performance improvement in :meth:`DataFrame.first_valid_index` and :meth:`DataFrame.last_valid_index` for extension array dtypes (:issue:`51549`)
• Performance improvement in :meth:`DataFrame.where` when cond is backed by an extension dtype (:issue:`51574`)
• Performance improvement in :meth:`MultiIndex.set_levels` and :meth:`MultiIndex.set_codes` when verify_integrity=True (:issue:`51873`)
• Performance improvement in :meth:`MultiIndex.sortlevel` when ascending is a list (:issue:`51612`)
• Performance improvement in :meth:`Series.combine_first` (:issue:`51777`)
• Performance improvement in :meth:`~arrays.ArrowExtensionArray.fillna` when array does not contain nulls (:issue:`51635`)
• Performance improvement in :meth:`~arrays.ArrowExtensionArray.isna` when array has zero nulls or is all nulls (:issue:`51630`)
• Performance improvement when parsing strings to boolean[pyarrow] dtype (:issue:`51730`)
• Performance improvement when searching an :class:`Index` sliced from other indexes (:issue:`51738`)
• Performance improvement in :func:`concat` (:issue:`52291`, :issue:`52290`)
• :class:`Period`'s default formatter (period_format) is now significantly (~twice) faster. This improves performance of str(Period), repr(Period), and :meth:`Period.strftime(fmt=None)`, as well as PeriodArray.strftime(fmt=None), PeriodIndex.strftime(fmt=None) and PeriodIndex.format(fmt=None). Finally, to_csv operations involving :class:`PeriodArray` or :class:`PeriodIndex` with default date_format are also significantly accelerated. (:issue:`51459`)
• Performance improvement accessing :attr:`arrays.IntegerArrays.dtype` & :attr:`arrays.FloatingArray.dtype` (:issue:`52998`)
• Performance improvement in :class:`MultiIndex` and multi-column operations (e.g. :meth:`DataFrame.sort_values`, :meth:`DataFrame.groupby`, :meth:`Series.unstack`) when index/column values are already sorted (:issue:`53806`)
• Performance improvement in :class:`Series` reductions (:issue:`52341`)
• Performance improvement in :func:`concat` when axis=1 and objects have different indexes (:issue:`52541`)
• Performance improvement in :func:`concat` when the concatenation axis is a :class:`MultiIndex` (:issue:`53574`)
• Performance improvement in :meth:`.DataFrameGroupBy.groups` (:issue:`53088`)
• Performance improvement in :meth:`DataFrame.isin` for extension dtypes (:issue:`53514`)
• Performance improvement in :meth:`DataFrame.loc` when selecting rows and columns (:issue:`53014`)
• Performance improvement in :meth:`Series.add` for pyarrow string and binary dtypes (:issue:`53150`)
• Performance improvement in :meth:`Series.corr` and :meth:`Series.cov` for extension dtypes (:issue:`52502`)
• Performance improvement in :meth:`Series.str.get_dummies` for pyarrow-backed strings (:issue:`53655`)
• Performance improvement in :meth:`Series.str.get` for pyarrow-backed strings (:issue:`53152`)
• Performance improvement in :meth:`Series.str.split` with expand=True for pyarrow-backed strings (:issue:`53585`)
• Performance improvement in :meth:`Series.to_numpy` when dtype is a numpy float dtype and na_value is np.nan (:issue:`52430`)
• Performance improvement in :meth:`~arrays.ArrowExtensionArray.astype` when converting from a pyarrow timestamp or duration dtype to numpy (:issue:`53326`)
• Performance improvement in :meth:`~arrays.ArrowExtensionArray.to_numpy` (:issue:`52525`)
• Performance improvement when doing various reshaping operations on :class:`arrays.IntegerArrays` & :class:`arrays.FloatingArray` by avoiding doing unnecessary validation (:issue:`53013`)
• Performance improvement when indexing with pyarrow timestamp and duration dtypes (:issue:`53368`)

Bug fixes

Categorical

• Bug in :meth:`Series.astype` with dtype="category" for nullable arrays with read-only null value masks (:issue:`53658`)
• Bug in :meth:`Series.map` , where the value of the na_action parameter was not used if the series held a :class:`Categorical` (:issue:`22527`).

Datetimelike

• :meth:`DatetimeIndex.map` with na_action="ignore" now works as expected. (:issue:`51644`)
• Bug in :class:`DateOffset` which had inconsistent behavior when multiplying a :class:`DateOffset` object by a constant (:issue:`47953`)
• Bug in :func:`date_range` when freq was a :class:`DateOffset` with nanoseconds (:issue:`46877`)
• Bug in :meth:`Timestamp.date`, :meth:`Timestamp.isocalendar`, :meth:`Timestamp.timetuple`, and :meth:`Timestamp.toordinal` were returning incorrect results for inputs outside those supported by the Python standard library's datetime module (:issue:`53668`)
• Bug in :meth:`Timestamp.round` with values close to the implementation bounds returning incorrect results instead of raising OutOfBoundsDatetime (:issue:`51494`)
• Bug in :meth:`arrays.DatetimeArray.map` and :meth:`DatetimeIndex.map`, where the supplied callable operated array-wise instead of element-wise (:issue:`51977`)
• Bug in constructing a :class:`Series` or :class:`DataFrame` from a datetime or timedelta scalar always inferring nanosecond resolution instead of inferring from the input (:issue:`52212`)
• Bug in parsing datetime strings with weekday but no day e.g. "2023 Sept Thu" incorrectly raising AttributeError instead of ValueError (:issue:`52659`)

Timedelta

• :meth:`TimedeltaIndex.map` with na_action="ignore" now works as expected (:issue:`51644`)
• Bug in :class:`TimedeltaIndex` division or multiplication leading to .freq of "0 Days" instead of None (:issue:`51575`)
• Bug in :class:`Timedelta` with Numpy timedelta64 objects not properly raising ValueError (:issue:`52806`)
• Bug in :meth:`Timedelta.round` with values close to the implementation bounds returning incorrect results instead of raising OutOfBoundsTimedelta (:issue:`51494`)
• Bug in :meth:`arrays.TimedeltaArray.map` and :meth:`TimedeltaIndex.map`, where the supplied callable operated array-wise instead of element-wise (:issue:`51977`)

Timezones

• Bug in :func:`infer_freq` that raises TypeError for Series of timezone-aware timestamps (:issue:`52456`)
• Bug in :meth:`DatetimeTZDtype.base` that always returns a NumPy dtype with nanosecond resolution (:issue:`52705`)

Numeric

• Bug in :class:`RangeIndex` setting step incorrectly when being the subtrahend with minuend a numeric value (:issue:`53255`)
• Bug in :meth:`Series.corr` and :meth:`Series.cov` raising AttributeError for masked dtypes (:issue:`51422`)
• Bug when calling :meth:`Series.kurt` and :meth:`Series.skew` on numpy data of all zero returning a python type instead of a numpy type (:issue:`53482`)
• Bug in :meth:`Series.mean`, :meth:`DataFrame.mean` with object-dtype values containing strings that can be converted to numbers (e.g. "2") returning incorrect numeric results; these now raise TypeError (:issue:`36703`, :issue:`44008`)
• Bug in :meth:`DataFrame.corrwith` raising NotImplementedError for pyarrow-backed dtypes (:issue:`52314`)
• Bug in :meth:`DataFrame.size` and :meth:`Series.size` returning 64-bit integer instead of int (:issue:`52897`)
• Bug in :meth:`Series.any`, :meth:`Series.all`, :meth:`DataFrame.any`, and :meth:`DataFrame.all` had the default value of bool_only set to None instead of False; this change should have no impact on users (:issue:`53258`)
• Bug in :meth:`Series.corr` and :meth:`Series.cov` raising AttributeError for masked dtypes (:issue:`51422`)
• Bug in :meth:`Series.median` and :meth:`DataFrame.median` with object-dtype values containing strings that can be converted to numbers (e.g. "2") returning incorrect numeric results; these now raise TypeError (:issue:`34671`)
• Bug in :meth:`Series.sum` converting dtype uint64 to int64 (:issue:`53401`)

Conversion

• Bug in :func:`DataFrame.style.to_latex` and :func:`DataFrame.style.to_html` if the DataFrame contains integers with more digits than can be represented by floating point double precision (:issue:`52272`)
• Bug in :func:`array` when given a datetime64 or timedelta64 dtype with unit of "s", "us", or "ms" returning :class:`PandasArray` instead of :class:`DatetimeArray` or :class:`TimedeltaArray` (:issue:`52859`)
• Bug in :meth:`ArrowDtype.numpy_dtype` returning nanosecond units for non-nanosecond pyarrow.timestamp and pyarrow.duration types (:issue:`51800`)
• Bug in :meth:`DataFrame.__repr__` incorrectly raising a TypeError when the dtype of a column is np.record (:issue:`48526`)
• Bug in :meth:`DataFrame.info` raising ValueError when use_numba is set (:issue:`51922`)
• Bug in :meth:`DataFrame.insert` raising TypeError if loc is np.int64 (:issue:`53193`)

Strings

Interval

• :meth:`pd.IntervalIndex.get_indexer` and :meth:`pd.IntervalIndex.get_indexer_nonunique` raising if target is read-only array (:issue:`53703`)

Indexing

• Bug in :meth:`DataFrame.__setitem__` losing dtype when setting a :class:`DataFrame` into duplicated columns (:issue:`53143`)
• Bug in :meth:`DataFrame.__setitem__` with a boolean mask and :meth:`DataFrame.putmask` with mixed non-numeric dtypes and a value other than NaN incorrectly raising TypeError (:issue:`53291`)
• Bug in :meth:`DataFrame.iloc` when using nan as the only element (:issue:`52234`)

Missing

• Bug in :meth:`DataFrame.interpolate` ignoring inplace when :class:`DataFrame` is empty (:issue:`53199`)
• Bug in :meth:`Series.interpolate` and :meth:`DataFrame.interpolate` failing to raise on invalid downcast keyword, which can be only None or "infer" (:issue:`53103`)
• Bug in :meth:`Series.interpolate` and :meth:`DataFrame.interpolate` with complex dtype incorrectly failing to fill NaN entries (:issue:`53635`)

MultiIndex

• Bug in :meth:`MultiIndex.set_levels` not preserving dtypes for :class:`Categorical` (:issue:`52125`)
• Bug in displaying a :class:`MultiIndex` with a long element (:issue:`52960`)

I/O

• :meth:`DataFrame.to_orc` now raising ValueError when non-default :class:`Index` is given (:issue:`51828`)
• :meth:`DataFrame.to_sql` now raising ValueError when the name param is left empty while using SQLAlchemy to connect (:issue:`52675`)
• Bug in :func:`json_normalize`, fix json_normalize cannot parse metadata fields list type (:issue:`37782`)
• Bug in :func:`read_csv` where it would error when parse_dates was set to a list or dictionary with engine="pyarrow" (:issue:`47961`)
• Bug in :func:`read_csv`, with engine="pyarrow" erroring when specifying a dtype with index_col (:issue:`53229`)
• Bug in :func:`read_hdf` not properly closing store after a IndexError is raised (:issue:`52781`)
• Bug in :func:`read_html`, style elements were read into DataFrames (:issue:`52197`)
• Bug in :func:`read_html`, tail texts were removed together with elements containing display:none style (:issue:`51629`)
• Bug in :func:`read_sql` when reading multiple timezone aware columns with the same column name (:issue:`44421`)
• Bug when writing and reading empty Stata dta files where dtype information was lost (:issue:`46240`)

Period

• :meth:`PeriodIndex.map` with na_action="ignore" now works as expected (:issue:`51644`)
• Bug in :class:`PeriodDtype` constructor failing to raise TypeError when no argument is passed or when None is passed (:issue:`27388`)
• Bug in :class:`PeriodDtype` constructor incorrectly returning the same normalize for different :class:`DateOffset` freq inputs (:issue:`24121`)
• Bug in :class:`PeriodDtype` constructor raising ValueError instead of TypeError when an invalid type is passed (:issue:`51790`)
• Bug in :func:`read_csv` not processing empty strings as a null value, with engine="pyarrow" (:issue:`52087`)
• Bug in :func:`read_csv` returning object dtype columns instead of float64 dtype columns with engine="pyarrow" for columns that are all null with engine="pyarrow" (:issue:`52087`)
• Bug in :meth:`arrays.PeriodArray.map` and :meth:`PeriodIndex.map`, where the supplied callable operated array-wise instead of element-wise (:issue:`51977`)
• Bug in incorrectly allowing construction of :class:`Period` or :class:`PeriodDtype` with :class:`CustomBusinessDay` freq; use :class:`BusinessDay` instead (:issue:`52534`)

Plotting

• Bug in :meth:`Series.plot` when invoked with color=None (:issue:`51953`)

Groupby/resample/rolling

• Bug in :meth:`DataFrame.resample` and :meth:`Series.resample` in incorrectly allowing non-fixed freq when resampling on a :class:`TimedeltaIndex` (:issue:`51896`)
• Bug in :meth:`DataFrame.resample` and :meth:`Series.resample` losing time zone when resampling empty data (:issue:`53664`)
• Bug in :meth:`DataFrameGroupBy.idxmin`, :meth:`SeriesGroupBy.idxmin`, :meth:`DataFrameGroupBy.idxmax`, :meth:`SeriesGroupBy.idxmax` return wrong dtype when used on empty DataFrameGroupBy or SeriesGroupBy (:issue:`51423`)
• Bug in weighted rolling aggregations when specifying min_periods=0 (:issue:`51449`)
• Bug in :meth:`DataFrame.groupby` and :meth:`Series.groupby`, where, when the index of the grouped :class:`Series` or :class:`DataFrame` was a :class:`DatetimeIndex`, :class:`TimedeltaIndex` or :class:`PeriodIndex`, and the groupby method was given a function as its first argument, the function operated on the whole index rather than each element of the index. (:issue:`51979`)
• Bug in :meth:`DataFrame.groupby` with column selection on the resulting groupby object not returning names as tuples when grouping by a list of a single element. (:issue:`53500`)
• Bug in :meth:`DataFrameGroupBy.agg` with lists not respecting as_index=False (:issue:`52849`)
• Bug in :meth:`DataFrameGroupBy.apply` causing an error to be raised when the input :class:`DataFrame` was subset as a :class:`DataFrame` after groupby ([['a']] and not ['a']) and the given callable returned :class:`Series` that were not all indexed the same. (:issue:`52444`)
• Bug in :meth:`DataFrameGroupBy.apply` raising a TypeError when selecting multiple columns and providing a function that returns np.ndarray results (:issue:`18930`)
• Bug in :meth:`GroupBy.groups` with a datetime key in conjunction with another key produced incorrect number of group keys (:issue:`51158`)
• Bug in :meth:`GroupBy.quantile` may implicitly sort the result index with sort=False (:issue:`53009`)
• Bug in :meth:`GroupBy.var` failing to raise TypeError when called with datetime64, timedelta64 or :class:`PeriodDtype` values (:issue:`52128`, :issue:`53045`)
• Bug in :meth:`SeriesGroupBy.nth` and :meth:`DataFrameGroupBy.nth` after performing column selection when using dropna="any" or dropna="all" would not subset columns (:issue:`53518`)
• Bug in :meth:`SeriesGroupBy.nth` and :meth:`DataFrameGroupBy.nth` raised after performing column selection when using dropna="any" or dropna="all" resulted in rows being dropped (:issue:`53518`)
• Bug in :meth:`SeriesGroupBy.sum` and :meth:`DataFrameGroupby.sum` summing np.inf + np.inf and (-np.inf) + (-np.inf) to np.nan (:issue:`53606`)

Reshaping

• Bug in :func:`concat` coercing to object dtype when one column has pa.null() dtype (:issue:`53702`)
• Bug in :func:`crosstab` when dropna=False would not keep np.nan in the result (:issue:`10772`)
• Bug in :func:`merge_asof` raising KeyError for extension dtypes (:issue:`52904`)
• Bug in :func:`merge_asof` raising ValueError for data backed by read-only ndarrays (:issue:`53513`)
• Bug in :meth:`DataFrame.agg` and :meth:`Series.agg` on non-unique columns would return incorrect type when dist-like argument passed in (:issue:`51099`)
• Bug in :meth:`DataFrame.combine_first` ignoring other's columns if other is empty (:issue:`53792`)
• Bug in :meth:`DataFrame.idxmin` and :meth:`DataFrame.idxmax`, where the axis dtype would be lost for empty frames (:issue:`53265`)
• Bug in :meth:`DataFrame.merge` not merging correctly when having MultiIndex with single level (:issue:`52331`)
• Bug in :meth:`DataFrame.stack` losing extension dtypes when columns is a :class:`MultiIndex` and frame contains mixed dtypes (:issue:`45740`)
• Bug in :meth:`DataFrame.stack` sorting columns lexicographically (:issue:`53786`)
• Bug in :meth:`DataFrame.transpose` inferring dtype for object column (:issue:`51546`)
• Bug in :meth:`Series.combine_first` converting int64 dtype to float64 and losing precision on very large integers (:issue:`51764`)

Sparse

• Bug in :class:`SparseDtype` constructor failing to raise TypeError when given an incompatible dtype for its subtype, which must be a numpy dtype (:issue:`53160`)
• Bug in :meth:`arrays.SparseArray.map` allowed the fill value to be included in the sparse values (:issue:`52095`)

ExtensionArray

• Bug in :class:`~arrays.ArrowExtensionArray` converting pandas non-nanosecond temporal objects from non-zero values to zero values (:issue:`53171`)
• Bug in :meth:`Series.quantile` for pyarrow temporal types raising ArrowInvalid (:issue:`52678`)
• Bug in :meth:`Series.rank` returning wrong order for small values with Float64 dtype (:issue:`52471`)
• Bug in :meth:`~arrays.ArrowExtensionArray.__iter__` and :meth:`~arrays.ArrowExtensionArray.__getitem__` returning python datetime and timedelta objects for non-nano dtypes (:issue:`53326`)
• Bug where the __from_arrow__ method of masked ExtensionDtypes(e.g. :class:`Float64Dtype`, :class:`BooleanDtype`) would not accept pyarrow arrays of type pyarrow.null() (:issue:`52223`)

Styler

• Bug in :meth:`Styler._copy` calling overridden methods in subclasses of :class:`Styler` (:issue:`52728`)

Metadata

• Fixed metadata propagation in :meth:`DataFrame.squeeze`, and :meth:`DataFrame.describe` (:issue:`28283`)
• Fixed metadata propagation in :meth:`DataFrame.std` (:issue:`28283`)

Other

• Bug in :class:`DataFrame` and :class:`Series` raising for data of complex dtype when NaN values are present (:issue:`53627`)
• Bug in :class:`FloatingArray.__contains__` with NaN item incorrectly returning False when NaN values are present (:issue:`52840`)
• Bug in :func:`api.interchange.from_dataframe` when converting an empty DataFrame object (:issue:`53155`)
• Bug in :func:`assert_almost_equal` now throwing assertion error for two unequal sets (:issue:`51727`)
• Bug in :func:`assert_frame_equal` checks category dtypes even when asked not to check index type (:issue:`52126`)
• Bug in :meth:`DataFrame.reindex` with a fill_value that should be inferred with a :class:`ExtensionDtype` incorrectly inferring object dtype (:issue:`52586`)
• Bug in :meth:`Series.align`, :meth:`DataFrame.align`, :meth:`Series.reindex`, :meth:`DataFrame.reindex`, :meth:`Series.interpolate`, :meth:`DataFrame.interpolate`, incorrectly failing to raise with method="asfreq" (:issue:`53620`)
• Bug in :meth:`Series.map` when giving a callable to an empty series, the returned series had object dtype. It now keeps the original dtype (:issue:`52384`)
• Bug in :meth:`Series.memory_usage` when deep=True throw an error with Series of objects and the returned value is incorrect, as it does not take into account GC corrections (:issue:`51858`)
• Fixed incorrect __name__ attribute of pandas._libs.json (:issue:`52898`)

Contributors