Skip to content

BUG: Catch overflow in both directions for checked add #14453

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 22, 2016
Merged

BUG: Catch overflow in both directions for checked add #14453

merged 1 commit into from
Oct 22, 2016

Conversation

gfyoung
Copy link
Member

@gfyoung gfyoung commented Oct 19, 2016

Port of #14324 after project name change.

@gfyoung
Copy link
Member Author

gfyoung commented Oct 19, 2016 •
edited
Loading

@jreback : Doesn't seem like Appveyor is picking up this PR.

@codecov-io
Copy link

codecov-io commented Oct 19, 2016 •
edited
Loading

Current coverage is 85.25% (diff: 68.75%)

Merging #14453 into master will decrease coverage by <.01%

@@             master     #14453   diff @@
==========================================
  Files           140        140          
  Lines         50640      50654    +14   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43175      43184     +9   
- Misses         7465       7470     +5   
  Partials          0          0

Powered by Codecov. Last update 65362aa...2dc49e4

@gfyoung gfyoung mentioned this pull request Oct 19, 2016
Closed
@jorisvandenbossche jorisvandenbossche added Bug Timedelta Timedelta data type labels Oct 19, 2016
@jreback jreback added this to the 0.19.1 milestone Oct 20, 2016
@jreback
Copy link
Contributor

jreback commented Oct 20, 2016

lgtm. can you add a whatsnew note. put in a new API Changes section in 0.19.1

@jorisvandenbossche
Copy link
Member

jorisvandenbossche commented Oct 20, 2016 •
edited
Loading

Please no api change section for 0.19.1. We put it in the bug section for 0.19.0, so then can put it here there as well I think (we maybe should have put it in the api changes section for 0.19.0 ..)

@gfyoung
BUG: Catch overflow in both directions for checked add
2dc49e4 
1) Add checks to ensure that add overflow does not
   occur both in the positive or negative directions.
2) Add benchmarks to ensure that operations involving
   this checked add function are significantly impacted.
@jreback
Copy link
Contributor

jreback commented Oct 20, 2016

bug fix section is fine

@jorisvandenbossche jorisvandenbossche merged commit 83a380c into pandas-dev:master Oct 22, 2016
@jorisvandenbossche
Copy link
Member

@gfyoung Thanks!

@gfyoung gfyoung deleted the add-overflow-bug branch October 22, 2016 15:03
@gfyoung
Copy link
Member Author

gfyoung commented Oct 24, 2016

@jreback , @jorisvandenbossche : So while it's good that we're catching underflow, when I try to integrate this check with tdi + tdi OR tdi + td, we run into issues:

>>> import pandas as pd
>>>
>>> tdi = pd.TimedeltaIndex([pd.NaT])
>>> td = pd.Timedelta('1 days')
>>>
>>> tdi - td   # Should return TimedeltaIndex([pd.NaT])
...
OverflowError: Overflow in int64 addition

This is because we are doing the checked add with the asi8 values, meaning that NaT becomes np.iinfo(np.int64).min, so the subtraction underflows as expected but not as we want. Not sure what the best way is to NOT do checked add for those elements whose value is NaN. Thoughts?

@jorisvandenbossche
Copy link
Member

Additionally mask for that in the checking code?

@gfyoung
Copy link
Member Author

gfyoung commented Oct 24, 2016

Hmmm...I suppose, though that will impact performance again 😢 . I can give it a shot anyways and see what happens.

jorisvandenbossche pushed a commit to jorisvandenbossche/pandas that referenced this pull request Nov 2, 2016
@gfyoung @jorisvandenbossche
[Backport 14453] BUG: Catch overflow in both directions for checked a…
c93ba9f 
…dd (pandas-dev#14453)

1) Add checks to ensure that add overflow does not
   occur both in the positive or negative directions.
2) Add benchmarks to ensure that operations involving
   this checked add function are significantly impacted.
(cherry picked from commit 83a380c)
gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 7, 2016
@gfyoung
BUG: Prevent addition overflow with TimedeltaIndex
8e21e60 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and TimeDelta). Follow-up to pandas-devgh-14453.
gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 7, 2016
@gfyoung
BUG: Prevent addition overflow with TimedeltaIndex
46909f0 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.
@gfyoung gfyoung mentioned this pull request Dec 7, 2016
Merged
gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 10, 2016
@gfyoung
BUG: Prevent addition overflow with TimedeltaIndex
ce48455 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.
gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 11, 2016
@gfyoung
BUG: Prevent addition overflow with TimedeltaIndex
7990eec 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.
gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 15, 2016
@gfyoung
BUG: Prevent addition overflow with TimedeltaIndex
e57f3bb 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.
gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 15, 2016
@gfyoung
BUG: Prevent addition overflow with TimedeltaIndex
d238fbb 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.
gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 15, 2016
@gfyoung
BUG: Prevent addition overflow with TimedeltaIndex
59ba12b 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.
gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 15, 2016
@gfyoung
BUG: Prevent addition overflow with TimedeltaIndex
4f8a12d 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.
gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 15, 2016
@gfyoung
BUG: Prevent addition overflow with TimedeltaIndex
7a3e54e 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.

In addition, move checked add function to core/algorithms.
gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 17, 2016
@gfyoung
BUG: Prevent addition overflow with TimedeltaIndex
a086db6 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.

In addition, move checked add function to core/algorithms.
jorisvandenbossche pushed a commit that referenced this pull request Dec 17, 2016
@gfyoung @jorisvandenbossche
BUG: Prevent addition overflow with TimedeltaIndex (#14816)
bdbebc4 
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.

In addition, move checked add function to core/algorithms.
ischurov pushed a commit to ischurov/pandas that referenced this pull request Dec 19, 2016
@gfyoung @ischurov
BUG: Prevent addition overflow with TimedeltaIndex (pandas-dev#14816)
ac5ec9d 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.

In addition, move checked add function to core/algorithms.
ShaharBental pushed a commit to ShaharBental/pandas that referenced this pull request Dec 26, 2016
@gfyoung @ShaharBental
BUG: Prevent addition overflow with TimedeltaIndex (pandas-dev#14816)
3398529 
Expands checked-add array addition introduced in
pandas-devgh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to pandas-devgh-14453.

In addition, move checked add function to core/algorithms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jreback jreback jreback approved these changes

Assignees
No one assigned
Labels
Bug Timedelta Timedelta data type
Projects
None yet
Milestone
0.19.1
Development

Successfully merging this pull request may close these issues.
4 participants
@gfyoung @codecov-io @jreback @jorisvandenbossche