Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: DataFrame.drop_duplicates method fails when a column with a list… #56958

Closed
wants to merge 1 commit into from

Conversation

manlattan
Copy link
Contributor

resolved #56784 in the file pandas/core/frame.py

Copy link
Member

@rhshadrach rhshadrach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

Comment on lines +6753 to +6754
To handle mutable objects such as list, convert the list column
to a tuple before using it in the subset.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not handle all cases. I think the note should start with something more general like:

This method requires data in the columns to be hashable.

... ])

>>> df['item_ids'] = df['item_ids'].apply(tuple)
>>> df.drop_duplicates(inplace=True)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inplace argument should be going away (hopefully soon); it should not be used here.

Comment on lines +6763 to +6765
>>> df['item_ids'] = df['item_ids'].apply(list)
number item_ids
0 1 [1, 2, 3]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line doesn't produce any output. If you want to show output, you can make it just df['items_ids'].apply(list) instead of assigning it back to the column.

@manlattan
Copy link
Contributor Author

Thank you so much for your suggestions @rhshadrach I'll make changes accordingly.

@mroeschke mroeschke added the duplicated duplicated, drop_duplicates label Jan 23, 2024
Copy link
Contributor

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

@github-actions github-actions bot added the Stale label Feb 23, 2024
@rhshadrach
Copy link
Member

This PR has gone stale. @manlattan - if you are interested in continuing, just comment here and we're happy to reopen!

@rhshadrach rhshadrach closed this Feb 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicated duplicated, drop_duplicates Stale
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: DataFrame.drop_duplicates method fails when a column with a list dtype is present
3 participants