Skip to content

ENH: column label filtering via regexes to work for numeric names #10384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 18 commits into from
Closed

ENH: column label filtering via regexes to work for numeric names #10384

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
ac58777
Update generic.py
cyrusmaher Jun 18, 2015
ac90352
Add test for regex filter on numeric column names
cyrusmaher Jul 3, 2015
b70b9c1
Add release note
cyrusmaher Jul 3, 2015
3a9934d
Add second regex test
cyrusmaher Jul 3, 2015
12d79e7
Fix regex filter for numeric columns
cyrusmaher Jun 18, 2015
ccc7490
Merge branch 'patch-1' of https://github.com/cyrusmaher/pandas into p…
cyrusmaher Jul 3, 2015
009422c
Update docs, test
cyrusmaher Jul 3, 2015
b46133f
Fix merge conflict
cyrusmaher Jul 3, 2015
5bd0a4b
# This is a combination of 2 commits.
cyrusmaher Jun 18, 2015
88a8e3e
Fix merge conflict
cyrusmaher Jul 3, 2015
49b607f
Fix merge conflict?
cyrusmaher Jul 3, 2015
2a9ddd1
Update generic.py
cyrusmaher Jun 18, 2015
0d3af4c
Add test for regex filter on numeric column names
cyrusmaher Jul 3, 2015
94626cc
Add release note
cyrusmaher Jul 3, 2015
3bb6d05
Add second regex test
cyrusmaher Jul 3, 2015
86d523a
Update docs, test
cyrusmaher Jul 3, 2015
f562f7f
Fix merge conflict
cyrusmaher Jul 3, 2015
d9c4523
Maybe this merge fix worked
cyrusmaher Jul 3, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/source/whatsnew/v0.17.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ New features

Other enhancements
^^^^^^^^^^^^^^^^^^

- `regex` argument to DataFrame.filter now handles numeric column names instead of raising an exception.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use double backticks here (and around DateFrame.filter)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add the issue number (this PR number) onto the end (see how the other issues are done)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

say instead of raising ValueError


.. _whatsnew_0170.api:

Backwards incompatible API changes
Expand Down
2 changes: 1 addition & 1 deletion pandas/core/generic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1928,7 +1928,7 @@ def filter(self, items=None, like=None, regex=None, axis=None):
return self.select(matchf, axis=axis_name)
elif regex:
matcher = re.compile(regex)
return self.select(lambda x: matcher.search(x) is not None,
return self.select(lambda x: matcher.search(str(x)) is not None,
axis=axis_name)
else:
raise TypeError('Must pass either `items`, `like`, or `regex`')
Expand Down
12 changes: 11 additions & 1 deletion pandas/tests/test_frame.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10741,7 +10741,7 @@ def test_filter(self):
idx = self.frame.index[0:4]
filtered = self.frame.filter(idx, axis='index')
expected = self.frame.reindex(index=idx)
assert_frame_equal(filtered,expected)
assert_frame_equal(filtered, expected)

# like
fcopy = self.frame.copy()
Expand All @@ -10755,6 +10755,16 @@ def test_filter(self):
df = DataFrame(0., index=[0, 1, 2], columns=[0, 1, '_A', '_B'])
filtered = df.filter(like='_')
self.assertEqual(len(filtered.columns), 2)

# regex with ints in column names
# from PR #10384
df = DataFrame(0., index=[0, 1, 2], columns=[0, 1, 'A1', 'B'])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add the issue number as a comment (this PR number since no associated issue)

filtered = df.filter(regex='^[0-9]+$')
self.assertEqual(len(filtered.columns), 2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do the test again with all number columns that are strings, e.g. ['0','1'...] (i think results should be the same)

on the comparision do

expected = DataFrame(.....)
assert_frame_equal(fitlered, expected)

IOW construct the expected manually

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use an assert_frame_equal here as well

you will need to explicty construct the expected, e.g. something like

expected = DataFrame(0,index=[0,1,2],columns=['A1','B'])

also change the test a bit to put the numerics not all at the beginning (e.g. put one in the middle or end)


expected = DataFrame(0., index=[0, 1, 2], columns=[0, 1, '0', '1'])
filtered = expected.filter(regex='^[0-9]+$') # shouldn't remove anything
self.assert_frame_equal(filtered, expected)

# pass in None
with assertRaisesRegexp(TypeError, 'Must pass'):
Expand Down