Skip to content

BUG: Series.resample fails on NaT index #39229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jan 26, 2021

Conversation

rhshadrach
Copy link
Member

@rhshadrach rhshadrach commented Jan 17, 2021

This will also resolve the core issue of #34998 (comment). cc @jorisvandenbossche

The added test:

    s = series.copy()
    s.index = PeriodIndex([NaT] * len(s), freq=freq)
    result = getattr(s.resample(freq), resample_method)()

currently fails for freq="M", where upsample is used instead of downsample leading to failure since the index is not unique. I can't tell if this is a bug or an operation that should raise. It also fails for DataFrame, so I think is unrelated to the issue here.

cc @jbrockmendel

@rhshadrach rhshadrach added Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Resample resample method Series Series data structure labels Jan 17, 2021
@rhshadrach rhshadrach requested a review from jreback January 21, 2021 02:42
else:
# if index is all NaT, return a single bin
data = [NaT]
bins = np.array([len(ax)])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait why do we have NaT in an index? e.g. why don't we have 0 bins?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Below this is the logic for when there are NaT along with non-NaT (lines 1712-1718 here):

        if nat_count > 0:
            # NaT handling as in pandas._lib.lib.generate_bins_dt64()
            # shift bins by the number of NaT
            bins += nat_count
            bins = np.insert(bins, 0, nat_count)
            binner = binner.insert(0, NaT)
            labels = labels.insert(0, NaT)

The added logic agrees with this in the case there are all NaT.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i c. can you unify this logic (maybe to a function)?

@jreback jreback added this to the 1.3 milestone Jan 26, 2021
@jreback jreback merged commit 779592d into pandas-dev:master Jan 26, 2021
@jreback
Copy link
Contributor

jreback commented Jan 26, 2021

thanks @rhshadrach

@rhshadrach rhshadrach deleted the resampler_series_nat branch January 26, 2021 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Resample resample method Series Series data structure
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: Series.resample fails on NaT PeriodIndex
3 participants