Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: standardize fill_value behavior across the API #15587

Closed
wants to merge 4 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions pandas/core/reshape.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@

import pandas.core.algorithms as algos
import pandas.algos as _algos
from pandas.core.missing import validate_fill_value

from pandas.core.index import MultiIndex, _get_na_value

Expand Down Expand Up @@ -405,6 +406,9 @@ def _slow_pivot(index, columns, values):


def unstack(obj, level, fill_value=None):
if fill_value:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually I would always pass this (we will make None an acceptable fill_value below)

validate_fill_value(fill_value, obj.values.dtype)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just pass obj.dtype, we never explictly call .values

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

obj may be a DataFrame AFAIK. I call values to get the numpy array here to consolidate the dtype (which means that e.g. a DataFrame with columns of mixed type will accept fill_value according to object rules). Is there a way to get this without accessing the underlying array directly?


if isinstance(level, (tuple, list)):
return _unstack_multiple(obj, level)

Expand Down
11 changes: 10 additions & 1 deletion pandas/tests/types/test_missing.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@
DatetimeIndex, TimedeltaIndex, date_range)
from pandas.types.dtypes import DatetimeTZDtype
from pandas.types.missing import (array_equivalent, isnull, notnull,
na_value_for_dtype)
na_value_for_dtype,
validate_fill_value)


def test_notnull():
Expand Down Expand Up @@ -301,3 +302,11 @@ def test_na_value_for_dtype():

for dtype in ['O']:
assert np.isnan(na_value_for_dtype(np.dtype(dtype)))


class TestValidateFillValue(tm.TestCase):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use a class, just create a function (and use parametrize)

# TODO: Fill out the test cases.
def test_validate_fill_value(self):
# validate_fill_value()
# import pdb; pdb.set_trace()
pass
32 changes: 31 additions & 1 deletion pandas/types/missing.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@
is_object_dtype,
is_integer,
_TD_DTYPE,
_NS_DTYPE)
_NS_DTYPE,
is_datetime64_any_dtype, is_float,
is_numeric_dtype, is_complex)
from datetime import datetime, timedelta
from .inference import is_list_like


Expand Down Expand Up @@ -391,3 +394,30 @@ def na_value_for_dtype(dtype):
elif is_bool_dtype(dtype):
return False
return np.nan


def validate_fill_value(value, dtype):
"""
Make sure the fill value is appropriate for the given dtype.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add in a Parameters, Returns, Raises section

"""
if not is_scalar(value):
raise TypeError('"fill_value" parameter must be '
'a scalar, but you passed a '
'"{0}"'.format(type(value).__name__))
elif not isnull(value):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually you already check that value is None is ok (just need a test to check!)

if is_numeric_dtype(dtype):
if not (is_float(value) or is_integer(value) or is_complex(value)):
raise TypeError('"fill_value" parameter must be '
'numeric, but you passed a '
'"{0}"'.format(type(value).__name__))
elif is_datetime64_any_dtype(dtype):
if not isinstance(value, (np.datetime64, datetime)):
raise TypeError('"fill_value" parameter must be a '
'datetime, but you passed a '
'"{0}"'.format(type(value).__name__))
elif is_timedelta64_dtype(dtype):
if not isinstance(value, (np.timedelta64, timedelta)):
raise TypeError('"value" parameter must be '
'a timedelta, but you passed a '
'"{0}"'.format(type(value).__name__))
# if object dtype, do nothing.