BUG: DataFrame.to_dict when orient=index data loss #22810

jameswinegar · 2018-09-23T01:26:47Z

closes DataFrame.to_dict can result in data loss when indexes are not unique #22801
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff

…not unique

…ame with non-unique index

pep8speaks · 2018-09-23T01:57:52Z

Hello @jameswinegar! Thanks for updating the PR.

There are no PEP8 issues in the file pandas/core/frame.py !
There are no PEP8 issues in the file pandas/tests/frame/test_convert_to.py !

Comment last updated on September 23, 2018 at 02:01 Hours UTC

codecov · 2018-09-23T03:37:40Z

Codecov Report

Merging #22810 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #22810      +/-   ##
==========================================
+ Coverage   92.19%   92.19%   +<.01%     
==========================================
  Files         169      169              
  Lines       50873    50913      +40     
==========================================
+ Hits        46904    46941      +37     
- Misses       3969     3972       +3

Flag	Coverage Δ
#multiple	`90.61% <100%> (ø)`	⬆️
#single	`42.3% <0%> (-0.02%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/frame.py	`97.11% <100%> (-0.1%)`	⬇️
pandas/core/internals/concat.py	`98% <0%> (-0.38%)`	⬇️
pandas/core/indexes/period.py	`93.31% <0%> (-0.35%)`	⬇️
pandas/core/indexes/datetimes.py	`95.74% <0%> (-0.04%)`	⬇️
pandas/core/indexing.py	`93.87% <0%> (-0.01%)`	⬇️
pandas/io/packers.py	`88.04% <0%> (ø)`	⬆️
pandas/core/dtypes/base.py	`100% <0%> (ø)`	⬆️
pandas/core/generic.py	`96.65% <0%> (ø)`	⬆️
pandas/core/arrays/datetimelike.py	`95.56% <0%> (ø)`	⬆️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 904af03...6aec074. Read the comment docs.

jreback

can you add a note in the api changes section about this for 0.24.0

jreback · 2018-09-23T12:16:09Z

pandas/core/frame.py

@@ -1163,8 +1163,13 @@ def to_dict(self, orient='dict', into=dict):
                           for k, v in zip(self.columns, np.atleast_1d(row)))
                    for row in self.values]
        elif orient.lower().startswith('i'):
-            return into_c((t[0], dict(zip(self.columns, t[1:])))
-                          for t in self.itertuples())
+            if self.index.unique().size == self.index.size:


you can just use self.index.is_unique

jreback · 2018-09-23T12:16:20Z

pandas/core/frame.py

+            if self.index.unique().size == self.index.size:
+                return into_c((t[0], dict(zip(self.columns, t[1:])))
+                              for t in self.itertuples())
+            else:


no need for an else!

pandas/tests/frame/test_convert_to.py

jreback

does this affect any other orient types?

can you add some equiv tests for .to_json? (or is this affected?)

jameswinegar · 2018-09-23T15:31:45Z

@jreback on .to_json #4376

jameswinegar · 2018-09-23T15:44:27Z

@jreback Where are the API changes documentation?

jreback · 2018-09-23T15:58:18Z

doc/source/whatsnew/v0.24.0.txt

@@ -191,6 +191,7 @@ Other Enhancements
 - :meth:`Series.resample` and :meth:`DataFrame.resample` have gained the :meth:`Resampler.quantile` (:issue:`15023`).
 - :meth:`Index.to_frame` now supports overriding column name(s) (:issue:`22580`).
 - New attribute :attr:`__git_version__` will return git commit sha of current build (:issue:`21295`).
+- :func:`~DataFrame.to_dict` with orient on index and a non-unique index will raise a ValueError instead of losing data (:issue:`22801`).


move to other api changes

jameswinegar · 2018-09-24T14:41:23Z

@jreback any more edits needed?

jreback · 2018-09-25T12:26:11Z

doc/source/whatsnew/v0.24.0.txt

@@ -804,6 +804,7 @@ Reshaping
 - Bug in :meth:`Series.replace` and meth:`DataFrame.replace` when dict is used as the ``to_replace`` value and one key in the dict is is another key's value, the results were inconsistent between using integer key and using string key (:issue:`20656`)
 - Bug in :meth:`DataFrame.drop_duplicates` for empty ``DataFrame`` which incorrectly raises an error (:issue:`20516`)
 - Bug in :func:`pandas.wide_to_long` when a string is passed to the stubnames argument and a column name is a substring of that stubname (:issue:`22468`)
+- Bug in :func:`DataFrame.to_dict` raises ``ValueError`` when used with ``orient='index'`` and a non-unique index instead of losing data (:issue:`22801`)


move to API Breaking changes

pandas/tests/frame/test_convert_to.py

jameswinegar · 2018-09-27T17:58:07Z

@jreback is that move good?

doc/source/whatsnew/v0.24.0.txt

pandas/tests/frame/test_convert_to.py

WillAyd

lgtm @jreback

jameswinegar · 2018-10-02T00:46:15Z

doc/source/whatsnew/v0.24.0.txt

@@ -847,7 +847,7 @@ Reshaping
 - Bug in :meth:`Series.replace` and meth:`DataFrame.replace` when dict is used as the ``to_replace`` value and one key in the dict is is another key's value, the results were inconsistent between using integer key and using string key (:issue:`20656`)
 - Bug in :meth:`DataFrame.drop_duplicates` for empty ``DataFrame`` which incorrectly raises an error (:issue:`20516`)
 - Bug in :func:`pandas.wide_to_long` when a string is passed to the stubnames argument and a column name is a substring of that stubname (:issue:`22468`)
-
+- Bug in :func:`merge` when merging ``datetime64[ns, tz]`` data that contained a DST transition (:issue:`18885`)


Resolved merge conflict

jreback

rebase and ping on green.

doc/source/whatsnew/v0.24.0.txt

jameswinegar · 2018-10-08T20:57:48Z

@jreback fixed. long wait last night for circleci

jreback · 2018-10-09T12:26:07Z

doc/source/whatsnew/v0.24.0.txt

+
+Bug in :func:`DataFrame.to_dict` raises ``ValueError`` when used with 
+``orient='index'`` and a non-unique index instead of losing data (:issue:`22801`)
+


can you add a mini-example here (just showing what kind of data frame will raise). the prior example was too long, but I think we need something.

Added an ipython block that will raise the error. @jreback

right but now the docs will not work. so, you can do this as an ipython block but you need an :okexcept: to handle the exception.

also show the df after defining it

Add example that will raise value error.

jreback · 2018-10-11T01:13:18Z

doc/source/whatsnew/v0.24.0.txt

+
+Bug in :func:`DataFrame.to_dict` raises ``ValueError`` when used with 
+``orient='index'`` and a non-unique index instead of losing data (:issue:`22801`)
+


right but now the docs will not work. so, you can do this as an ipython block but you need an :okexcept: to handle the exception.

also show the df after defining it

jreback · 2018-10-11T01:47:29Z

doc/source/whatsnew/v0.24.0.txt

+``orient='index'`` and a non-unique index instead of losing data (:issue:`22801`)
+
+.. ipython:: python
+:okexcept:


line this up as below, e.g.

.. ipython:: python :okexcept: df = ......

jreback

one more doc correction, ping on green.

jreback · 2018-10-11T13:09:10Z

thanks @jameswinegar

jameswinegar added 2 commits September 22, 2018 20:24

Raise ValueError on DataFrame.to_dict when orient=index and index is …

8164725

…not unique

change to numpy based size and use methods. Fix test to create datafr…

616db89

…ame with non-unique index

jameswinegar added 2 commits September 22, 2018 21:00

fix pep8

9949fb7

fix reference to dataframe as self

8522e13

jreback requested changes Sep 23, 2018

View reviewed changes

jreback added Reshaping Concat, Merge/Join, Stack/Unstack, Explode Error Reporting Incorrect or improved errors from pandas labels Sep 23, 2018

jreback requested changes Sep 23, 2018

View reviewed changes

adjust for @jreback comments

df58b9d

jameswinegar added 2 commits September 23, 2018 10:49

Merge branch 'master' of https://github.com/pandas-dev/pandas

2de5e0a

add documentation and get origin to avoid merge conflict

51e42da

jreback reviewed Sep 23, 2018

View reviewed changes

jameswinegar added 2 commits September 23, 2018 11:05

move documentation and reword for clarity and consistency

0a49ffc

resolve merge conflict

abeda10

jreback requested changes Sep 25, 2018

View reviewed changes

Update v0.24.0.txt

6eb2e5a

WillAyd requested changes Sep 27, 2018

View reviewed changes

doc/source/whatsnew/v0.24.0.txt Show resolved Hide resolved

pandas/tests/frame/test_convert_to.py Show resolved Hide resolved

WillAyd approved these changes Oct 1, 2018

View reviewed changes

jameswinegar and others added 3 commits October 1, 2018 19:42

resolve merge conflict

317555b

Merge branch 'master' into master

f457cd4

Merge branch 'master' of https://github.com/jameswinegar/pandas

0da33da

jameswinegar commented Oct 2, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Oct 7, 2018

jreback requested changes Oct 7, 2018

View reviewed changes

doc/source/whatsnew/v0.24.0.txt Outdated Show resolved Hide resolved

jameswinegar added 2 commits October 7, 2018 20:03

Merge branch 'master' of https://github.com/pandas-dev/pandas

4c57b78

less verbose whatsnew

fb2eff0

jreback reviewed Oct 9, 2018

View reviewed changes

Update v0.24.0.txt

7a4240e

Add example that will raise value error.

jreback requested changes Oct 11, 2018

View reviewed changes

Update v0.24.0.txt

a5e8945

jreback reviewed Oct 11, 2018

View reviewed changes

jreback approved these changes Oct 11, 2018

View reviewed changes

Update v0.24.0.txt

6aec074

jreback merged commit c8ce3d0 into pandas-dev:master Oct 11, 2018

tm9k1 pushed a commit to tm9k1/pandas that referenced this pull request Nov 19, 2018

BUG: DataFrame.to_dict when orient=index data loss (pandas-dev#22810)

19ca934

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: DataFrame.to_dict when orient=index data loss #22810

BUG: DataFrame.to_dict when orient=index data loss #22810

jameswinegar commented Sep 23, 2018

pep8speaks commented Sep 23, 2018 •

edited

Loading

codecov bot commented Sep 23, 2018 •

edited

Loading

jreback left a comment

jreback Sep 23, 2018

jreback Sep 23, 2018

jreback left a comment

jameswinegar commented Sep 23, 2018

jameswinegar commented Sep 23, 2018

jreback Sep 23, 2018

jameswinegar commented Sep 24, 2018

jreback Sep 25, 2018

jameswinegar commented Sep 27, 2018

WillAyd left a comment

jameswinegar Oct 2, 2018

jreback left a comment

jameswinegar commented Oct 8, 2018

jreback Oct 9, 2018

jameswinegar Oct 10, 2018

jreback Oct 11, 2018

jreback Oct 11, 2018

jreback Oct 11, 2018

jreback left a comment

jreback commented Oct 11, 2018


		Bug in :func:`DataFrame.to_dict` raises ``ValueError`` when used with
		``orient='index'`` and a non-unique index instead of losing data (:issue:`22801`)

BUG: DataFrame.to_dict when orient=index data loss #22810

BUG: DataFrame.to_dict when orient=index data loss #22810

Conversation

jameswinegar commented Sep 23, 2018

pep8speaks commented Sep 23, 2018 • edited Loading

Comment last updated on September 23, 2018 at 02:01 Hours UTC

codecov bot commented Sep 23, 2018 • edited Loading

Codecov Report

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jameswinegar commented Sep 23, 2018

jameswinegar commented Sep 23, 2018

Choose a reason for hiding this comment

jameswinegar commented Sep 24, 2018

Choose a reason for hiding this comment

jameswinegar commented Sep 27, 2018

WillAyd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jameswinegar commented Oct 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback commented Oct 11, 2018

pep8speaks commented Sep 23, 2018 •

edited

Loading

codecov bot commented Sep 23, 2018 •

edited

Loading