MNT: Bump dev pin on NumPy #60987

rhshadrach · 2025-02-22T21:22:35Z

closes #xxxx (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

…_numpy

doc/source/user_guide/basics.rst

…_numpy

rhshadrach · 2025-03-06T22:40:39Z

pandas/core/base.py

        >>> s.argmax()
-        2
+        np.int64(2)
        >>> s.argmin()
-        0


There are a few of these where I'm wondering if we should be returning Python scalars instead of NumPy. Should issues be opened for these?

cc @pandas-dev/pandas-core

I think generally we always want to return Python scalars (IIRC we got a lot of issues about this in iteration and iteration-like APIs in the past)

Even just wrapping the result of Series._reduce with maybe_box_naive breaks 692 tests. From a cursory look, they're tests that are expecting a NumPy scalar back. A lot however are something like op(data).any().any() so that they will work with DataFrame and Series. I plan to bring this up in the next dev meeting.

I agree we should always return Python scalars. I'm surprised at the amount of failures that expect NumPy scalars

I'd think you need a deprecation on this, because people may have code that depends on the result being a numpy scalar. I think that the tests we have in pandas-stubs for typing may depend on this.

We could put it up behind a future option, maybe something like always_return_python_scalars

mroeschke · 2025-03-07T23:21:34Z

pandas/core/algorithms.py

@@ -415,7 +415,7 @@ def unique(values):

    >>> pd.unique(pd.array([1 + 1j, 2, 3]))
    <NumpyExtensionArray>
-    [(1+1j), (2+0j), (3+0j)]
+    [np.complex128(1+1j), np.complex128(2+0j), np.complex128(3+0j)]


I merged #54268 a while back to make these reprs render like the Python scalars/pre numpy 2.0. It appears that PR didn't touch all the relevant repr methods

I'll do a precursor to fix the reprs for arrays, and then revert some of the outputs here.

Your comment only applies to EA reprs?

I would say any repr in pandas shouldn't be showing the NEP 51 style repr for scalars.

I would say any repr in pandas shouldn't be showing the NEP 51 style repr for scalars.

The main motivation behind NEP 51 is that the Python numerical types behave differently from the NumPy scalars.

They give a few examples such as

Python programmers sometimes write obj is True and will surprised when an object that shows as True fails to pass the test. It is much easier to understand this behavior when the value is shown as np.True_.

So I think we should be showing the NEP 51 style repr for scalars.

@mroeschke you have approved this PR so I assume that was a typo?

…_numpy

rhshadrach · 2025-03-20T09:57:54Z

@mroeschke @WillAyd - it will be a significant change to start returning Python objects instead of NumPy across the board. Do we want to make that (in separate PRs) before merging this?

Dr-Irv · 2025-03-20T14:50:28Z

pandas/core/arrays/datetimelike.py

+        print(offset1, offset2)
+        print(type(offset1), type(offset2))


print in pandas??

Doh, thanks.

Dr-Irv · 2025-03-20T14:51:45Z

pandas/core/base.py

        >>> s.argmax()
-        2
+        np.int64(2)
        >>> s.argmin()
-        0


I'd think you need a deprecation on this, because people may have code that depends on the result being a numpy scalar. I think that the tests we have in pandas-stubs for typing may depend on this.

Dr-Irv · 2025-03-20T14:53:24Z

requirements-dev.txt

@@ -14,7 +14,7 @@ pytest-localserver
 PyQt5>=5.15.9
 coverage
 python-dateutil
-numpy<2
+numpy<3


At what point do we stop numpy < 2 support?

Spec 0 says Q3 2025. I would suggest pandas 4.0, but perhaps it could be earlier. In any case, this doesn't seem like the best place to discuss this at length.

WillAyd · 2025-03-20T19:27:32Z

If it is a significant effort then I would say as long as we don't introduce new NumPy scalars in return values then we are good. We can leave it to separate exercises to hack away at that

rhshadrach · 2025-03-20T20:31:40Z

as long as we don't introduce new NumPy scalars in return values then we are good

Right - this PR does not introduce any new instances of NumPy scalars, just adjusts the docstrings with the new repr as of NumPy 2.0.

WillAyd · 2025-03-20T21:04:03Z

So prior to NumPy 2.0 these returned Python scalars but now they return NumPy scalars?

rhshadrach · 2025-03-20T21:05:04Z

No, the same object is being returned - always a NumPy scalar. Only the repr is changing when you go from NumPy < 2 to NumPy >= 2.

mroeschke

LGTM

simonjayhawkins · 2025-03-27T11:49:44Z

pandas/core/arrays/base.py

@@ -1962,7 +1962,7 @@ def _formatter(self, boxed: bool = False) -> Callable[[Any], str | None]:
        ...         return lambda x: "*" + str(x) + "*" if boxed else repr(x) + "*"
        >>> MyExtensionArray(np.array([1, 2, 3, 4]))
        <MyExtensionArray>
-        [1*, 2*, 3*, 4*]
+        [np.int64(1)*, np.int64(2)*, np.int64(3)*, np.int64(4)*]


NEP 51 is for the output of numpy scalars to help users distinguish the NumPy scalars from the Python builtin types and clarify their behavior and that array representation will not be affected since it already includes the dtype= when necessary.

So should probably change the example here to not use the NEP 51 repr in the output?

I agree with Simon. Historically I don't think we have ever favored exposing NumPy scalars directly to end users. If that has happened as an implementation detail that is one thing, but explicitly showing that to an end user runs counter to what we have done in the psat

simonjayhawkins · 2025-03-27T12:47:59Z

pandas/core/arrays/interval.py

@@ -1775,7 +1775,8 @@ def to_tuples(self, na_tuple: bool = True) -> np.ndarray:
        [(0, 1], (1, 2]]
        Length: 2, dtype: interval[int64, right]
        >>> idx.to_tuples()
-        array([(0, 1), (1, 2)], dtype=object)
+        array([(np.int64(0), np.int64(1)), (np.int64(1), np.int64(2))],
+              dtype=object)


In https://github.com/pandas-dev/pandas/pull/60987/files#r2016346826 I commented because the dtype of the array was the same as the dtype of the scalars.

Thinking some more, we currently have...

pd.Series([123,"123"]) # 0 123 # 1 123 # dtype: object

so for consistency, even object arrays should probably not show the NEP51 repr?

I don't think that's an option here, this is what happens when you call str on a tuple of NumPy scalars.

print(str((np.int64(0), np.int64(1)))) # (np.int64(0), np.int64(1)) arr = pd.array([(np.int64(0), np.int64(1)), (np.int64(1), np.int64(2))], dtype=object) print(type(arr[0]), str(arr[0])) # <class 'tuple'> (np.int64(0), np.int64(1))

https://peps.python.org/pep-3140/

seems reasonable however extending the example I gave for the strings in an object array...

s = pd.Series([123, "123", ("123", "123")]) print(s) print(s[2]) # 0 123 # 1 123 # 2 (123, 123) # dtype: object # ('123', '123')

however, we don't currently display the repr for strings in a collection either when the array is displayed?

my bad, the numpy array of values does show the repr for the strings in a tuple.

s.values # array([123, '123', ('123', '123')], dtype=object)

simonjayhawkins

Thanks @rhshadrach lgtm

rhshadrach · 2025-03-31T21:24:42Z

@Dr-Irv - friendly ping

Dr-Irv · 2025-04-03T14:46:32Z

@Dr-Irv - friendly ping

Sorry - I was out the past 5 days! Will look now

mroeschke · 2025-04-03T15:30:32Z

Thanks @rhshadrach

rhshadrach added 6 commits February 22, 2025 16:22

MNT: Bump dev pin on NumPy

658c4d5

Fix type-hints

e424a96

Merge branch 'main' of https://github.com/pandas-dev/pandas into bump…

912bc1b

…_numpy

Docs fixup

e75abfe

fixups

856a52f

Fixup

a8304b6

rhshadrach commented Mar 1, 2025

View reviewed changes

doc/source/user_guide/basics.rst Outdated Show resolved Hide resolved

rhshadrach added 3 commits March 1, 2025 17:33

Fixup

41265f3

Merge branch 'main' of https://github.com/pandas-dev/pandas into bump…

9c2f5ac

…_numpy

doctest fixups

f6a2330

rhshadrach commented Mar 6, 2025

View reviewed changes

More doc fixes

83e2644

mroeschke reviewed Mar 7, 2025

View reviewed changes

rhshadrach marked this pull request as draft March 7, 2025 23:39

rhshadrach mentioned this pull request Mar 7, 2025

BUG: Printing float16 with NumPy>=2.0 gives overflow warning #61029

Closed

rhshadrach added 3 commits March 19, 2025 13:42

Merge branch 'main' of https://github.com/pandas-dev/pandas into bump…

8fd50b4

…_numpy

Some reverts

f7ef882

Avoid float16

347b865

rhshadrach marked this pull request as ready for review March 20, 2025 09:56

Dr-Irv requested changes Mar 20, 2025

View reviewed changes

Cleanup

17ec834

rhshadrach requested a review from mroeschke March 26, 2025 15:13

mroeschke added this to the 3.0 milestone Mar 26, 2025

mroeschke added the Dependencies Required and optional dependencies label Mar 26, 2025

mroeschke approved these changes Mar 26, 2025

View reviewed changes

simonjayhawkins reviewed Mar 27, 2025

View reviewed changes

Remove repr from MyExtensionArray

65e1374

rhshadrach requested review from WillAyd, simonjayhawkins and Dr-Irv March 29, 2025 12:43

simonjayhawkins approved these changes Mar 29, 2025

View reviewed changes

Fixup

14d34a7

Dr-Irv approved these changes Apr 3, 2025

View reviewed changes

mroeschke merged commit 1f1e13f into pandas-dev:main Apr 3, 2025
42 checks passed

rhshadrach deleted the bump_numpy branch April 3, 2025 20:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MNT: Bump dev pin on NumPy #60987

MNT: Bump dev pin on NumPy #60987

rhshadrach commented Feb 22, 2025

rhshadrach Mar 6, 2025

mroeschke Mar 7, 2025

rhshadrach Mar 8, 2025

WillAyd Mar 10, 2025

Dr-Irv Mar 20, 2025

rhshadrach Mar 20, 2025

mroeschke Mar 7, 2025

rhshadrach Mar 7, 2025

mroeschke Mar 8, 2025

simonjayhawkins Mar 27, 2025

rhshadrach commented Mar 20, 2025 •

edited

Loading

Dr-Irv Mar 20, 2025

rhshadrach Mar 20, 2025

Dr-Irv Mar 20, 2025

Dr-Irv Mar 20, 2025

rhshadrach Mar 20, 2025

WillAyd commented Mar 20, 2025

rhshadrach commented Mar 20, 2025

WillAyd commented Mar 20, 2025

rhshadrach commented Mar 20, 2025 •

edited

Loading

mroeschke left a comment

simonjayhawkins Mar 27, 2025

WillAyd Mar 27, 2025

rhshadrach Mar 29, 2025

simonjayhawkins Mar 27, 2025

rhshadrach Mar 29, 2025

rhshadrach Mar 29, 2025

simonjayhawkins Mar 29, 2025

simonjayhawkins Mar 29, 2025

simonjayhawkins left a comment

rhshadrach commented Mar 31, 2025

Dr-Irv commented Apr 3, 2025

mroeschke commented Apr 3, 2025

MNT: Bump dev pin on NumPy #60987

MNT: Bump dev pin on NumPy #60987

Conversation

rhshadrach commented Feb 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhshadrach commented Mar 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Mar 20, 2025

rhshadrach commented Mar 20, 2025

WillAyd commented Mar 20, 2025

rhshadrach commented Mar 20, 2025 • edited Loading

mroeschke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonjayhawkins left a comment

Choose a reason for hiding this comment

rhshadrach commented Mar 31, 2025

Dr-Irv commented Apr 3, 2025

mroeschke commented Apr 3, 2025

rhshadrach commented Mar 20, 2025 •

edited

Loading

rhshadrach commented Mar 20, 2025 •

edited

Loading