BUG: pd.Series.isnumeric() doesn't work on decimal value strings

[sf-dcp]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

df = pd.DataFrame({"string_values": ["1", "1.0", "1.1"]})
df.string_values.str.isnumeric()

Issue Description

The series method .isnumeric() only works on integer strings. If a string number is decimal, it will return False. When running the example below, the following is returned:

This is the docs description for the method:

Expected Behavior

Running the method on decimal strings should return True.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.12.8
python-bits : 64
OS : Linux
OS-release : 5.15.49-linuxkit-pr
Version : #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : C.UTF-8

pandas : 2.2.3
numpy : 2.2.1
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 24.3.1
Cython : None
sphinx : None
IPython : 8.31.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.5
lxml.etree : 5.3.0
matplotlib : 3.10.0
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : 2.9.10
pymysql : None
pyarrow : 18.1.0
pyreadstat : None
pytest : 8.3.4
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.15.1
sqlalchemy : 2.0.37
tables : None
tabulate : 0.9.0
xarray : None
xlrd : 2.0.1
xlsxwriter : None
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None

[akj2018]

Hi @sf-dcp,

This is not a bug in pd.Series.isnumeric() but expected behavior. As per documentation for str.isnumeric(), . (decimal point) is not a numeric character, so strings with decimal points like "1.1" fail the isnumeric() check.

str.isnumeric() method checks if:

1. All characters in the string are numeric characters.
2. There is at least one character in the string.

Numeric characters are those with the Unicode property values:

• Numeric_Type=Digit (e.g., "1", "2", "3")
• Numeric_Type=Decimal (e.g., standard decimal digits like "1", "2")
• Numeric_Type=Numeric (e.g., fractions like "⅕", Roman numerals, etc.)

Therefore, . (decimal character) does not have the Unicode numeric property; it's categorized as punctuation.

Solution 01 - Using regex

import re

def is_numeric(string):
    pattern = r'^-?\d+(\.\d+)?$'
    return bool(re.match(pattern, string))

Note:

• correctly identifies valid numeric strings like "1", "-2.5" and "0.0".
• rejects invalid strings like "1.0.0", "abc", or empty strings.
• Does not works for scientific notations, "1e10" returns False

Solution 02 - Using float()

def is_numeric(string):
    try:
        float(string)
        return True
    except ValueError:
        return False

Note:

• correctly identifies valid numeric strings like "1", "-2.5", "1e10" (scientific notation), and "0.0"
• rejects invalid strings like "1.0.0", "abc", or empty strings.

Performance Consideration

• Regex: Slower due to the regex engine's overhead of parsing and matching patterns.
• Float-based: Faster, as it directly leverages Python’s built-in C-optimized parsing.

# Test dataset (mix of numeric and non-numeric strings)
data = ["1", "123.45", "-987.65", "0", "abc", "123.45.67"] * 1_000_000

• Regex time: 5.59 seconds
• Float time: 2.23 seconds

[sf-dcp]

Hi @akj2018, thanks for the prompt reply and suggestions. I think the method name is a bit misleading then where I would expect a string number (with a dot or a negative integer string) to be returned True. But I also understand the logic behind it. At a minimum, I would recommend updating the method docs with a note or warning calling out the method's behavior when it comes to decimal numbers and/or negative numbers. Again, thanks for clarifying.

[rhshadrach]

Thanks for the report! This mirrors Python behavior:

print("1.2".isnumeric())
# False

The first line of the docstring states:

Check whether all characters in each string are numeric.

Are you saying there might be some confusion as to whether . is a numeric character?

[sf-dcp]

@rhshadrach, yep, I may be biased here but I can interpret . being numeric like in a decimal number :) I believe an explicit message about numeric values other than integers (negative and decimal string values) would be beneficial for at least a portion of pandas users.

[rhshadrach]

I'd be okay with this addition. PRs are welcome!

[akj2018]

take

[JyotiP24]

Hi,
I'd like to take on this issue and contribute to improving the documentation to explicitly clarify the behavior of pd.Series.str.isnumeric() regarding decimal and negative numbers. I'll ensure the updated docs provide clear examples and explanations for users. Please let me know if there are any additional points you'd like me to cover.

[akj2018]

Hi @JyotiP24 ,
I am currently working on the issue and will be making a PR tomorrow.
Thanks

[prakhar-bip]

what if we try this code ?

def is_numeric(string):
    if (string.isnumeric()):
        return True;

    else:
        try:
            float(string)
            return True;
        except ValueError:
            return False

[arnav1209]

I noticed that pd.Series.isnumeric() currently returns False for '3.14' due to the behavior of Python’s str.isnumeric(). Would the intended fix be to have isnumeric() return True for strings that represent valid decimal numbers? Also, should we extend this to handle signs (like -3.14) or scientific notation (like 1e3), or keep it simple?

[rhshadrach]

Would the intended fix be to have isnumeric() return True for strings that represent valid decimal numbers? Also, should we extend this to handle signs (like -3.14) or scientific notation (like 1e3), or keep it simple?

No, pandas' isnumeic should match Python's isnumeric.

[pranavgudagi]

Is this issue still open i want to contribute to this issue .Please assign me this issue .

[akj2018]

Hi @pranavgudagi ,
I am currently working on the issue.
Thanks

[WalkingDevFlag]

@forbid_nonstring_types(['bytes'])
def isnumeric(self, extended=False):
    if extended:
        regex = r'^[+-]?(\d+([.]\d*)?([eE][+-]?\d+)?|[.]\d+([eE][+-]?\d+)?)$'
        return self._data.str.match(regex, na=False)
    return self._data._str_map(
        fname="isnumeric", dtype=np.dtype("bool"), na_value=False
    )

would this be a good solution to handling decimal strings?

[Varun270]

@akj2018 Are You still working on this?

[Anurag-Varma]

@arnav1209

I noticed that pd.Series.isnumeric() currently returns False for '3.14' due to the behavior of Python’s str.isnumeric(). Would the intended fix be to have isnumeric() return True for strings that represent valid decimal numbers? Also, should we extend this to handle signs (like -3.14) or scientific notation (like 1e3), or keep it simple?

As mentioned, we should not change the code, this bug is a documentation fixing issue

No, pandas' isnumeic should match Python's isnumeric.

[Anurag-Varma]

Hey @akj2018

It's been 2 weeks, can you confirm if you are still working on it or if I can take the issue ?

[akj2018]

Hi @Anurag-Varma,

Yes, I am still working on it. Due to health related concerns, I was not able to complete it. I will make a PR today without further delay.

In future, please me mindful to wait for a while before taking an action.

Thanks

[Anurag-Varma]

Hi @Anurag-Varma,

Yes, I am still working on it. Due to health related concerns, I was not able to complete it. I will make a PR today without further delay.

In future, please me mindful to wait for a while before taking an action.

Thanks

It's been 3 weeks since your last update on PR and 2 weeks ago you have mentioned that you are still working. I didn't find any updates on your remote repository in your account related to this issue so I just took it. But if you are still working then you can continue and send a PR.

[akj2018]

@Anurag-Varma Yes, because I haven't updated the remote with my local changes.

Kindly refer the contribution guide in such situations, which mentions to ask the assigne along with period of inactivity.

https://pandas.pydata.org/docs/development/contributing.html#finding-an-issue-to-contribute-to

Rest assured that I will making a PR today so there is no further delay.