-
-
Notifications
You must be signed in to change notification settings - Fork 18.5k
ENH: Add Timedelta Support to JSON Reader with orient=table (#21140) #21827
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 4 commits
7be068b
7f1336c
b0150e4
ed0e1fe
6842582
0f28fd0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,7 +6,7 @@ | |
import warnings | ||
|
||
import pandas._libs.json as json | ||
from pandas import DataFrame | ||
from pandas import DataFrame, to_timedelta | ||
from pandas.api.types import CategoricalDtype | ||
import pandas.core.common as com | ||
from pandas.core.dtypes.common import ( | ||
|
@@ -163,7 +163,7 @@ def convert_json_field_to_pandas_type(field): | |
elif typ == 'boolean': | ||
return 'bool' | ||
elif typ == 'duration': | ||
return 'timedelta64' | ||
return 'timedelta64[ns]' | ||
elif typ == 'datetime': | ||
if field.get('tz'): | ||
return 'datetime64[ns, {tz}]'.format(tz=field['tz']) | ||
|
@@ -306,10 +306,9 @@ def parse_table_schema(json, precise_float): | |
raise NotImplementedError('table="orient" can not yet read timezone ' | ||
'data') | ||
|
||
# No ISO constructor for Timedelta as of yet, so need to raise | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. was this not hit in tests prior to this PR? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should have been covered by |
||
if 'timedelta64' in dtypes.values(): | ||
raise NotImplementedError('table="orient" can not yet read ' | ||
'ISO-formatted Timedelta data') | ||
for col, dtype in dtypes.items(): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this block necessary? Assumed the subsequent There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So, support for iso-format timedeltas should be added to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No just go with what jreback said for now. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. hmm, are you sure you need to do this? I believe There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For now delta = pd.Timedelta(1e9).isoformat()
pd.DataFrame([delta]).astype('timedelta64[ns]') results in error. |
||
if dtype == 'timedelta64[ns]': | ||
df[col] = to_timedelta(df[col]) | ||
|
||
df = df.astype(dtypes) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there any conflict between this and the loop above? Wondering if we should be removing any timedelta calls from the |
||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might have missed this on previous comment but there's a small typo here.
parse
->parses
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@WillAyd I've pushed a fix, but now there is a merge conflict. How should I deal with it on github?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Better to deal locally and re-push. Assuming you have things set up as mentioned in the pandas contributing guide do this on your local branch
You'll probably get a merge conflict there, so fix that up and do:
Everything should be resolved then so re-push after that