Skip to content

BUG: read_hdf bad filtering in case of categorical string columns #39351

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
bb2d803
BUG: fix case of a category value which isn't exists (#39189)
nofarmish Jan 22, 2021
f9be625
BUG: add UT to conver_value for this use case (#39189)
nofarmish Jan 23, 2021
aa90441
BUG: change style with pre-commit (#39189)
nofarmish Jan 23, 2021
e8ca3fc
BUG: add a whatsnew record (#39189)
nofarmish Jan 23, 2021
b5ded49
Trigger Build
nofarmish Jan 23, 2021
0cb8ad7
BUG: check for tests (#39189)
nofarmish Jan 23, 2021
8284e0b
BUG: remove spaces (#39189)
nofarmish Jan 23, 2021
9773aaa
BUG: remove whatsnew (#39189)
nofarmish Jan 23, 2021
4281ef0
BUG: remove tests(#39189)
nofarmish Jan 23, 2021
7178757
BUG: add whats new (#39189)
nofarmish Jan 23, 2021
ca9420e
BUG: check tests (#39189)
nofarmish Jan 23, 2021
27728f0
remove incorrect xfails (#39354)
jbrockmendel Jan 24, 2021
11ee222
TST: fix fillna test for PandasArray (#39361)
jbrockmendel Jan 24, 2021
64cfcf4
TST: fix incorrect extension test (#39344)
jbrockmendel Jan 24, 2021
fc266a8
TST: fix xfailed tests for IntegerArray, FloatingArray (#39350)
jbrockmendel Jan 24, 2021
a1e8b59
BUG+DOC: Recent Styler Enhancements (#39317)
attack68 Jan 24, 2021
9e47ff7
BUG: incorrect casting ints to Period in GroupBy.agg (#39362)
jbrockmendel Jan 24, 2021
caf81fa
BUG: DataFrame.__setitem__ not raising ValueError when rhs is df and …
phofl Jan 24, 2021
37b5800
TST: tighten PandasArray tests (#39370)
jbrockmendel Jan 24, 2021
8d7ff93
CI: fix PandasArray test (#39384)
jbrockmendel Jan 25, 2021
88bea62
REF: move PandasDtype to dtypes.dtypes (#39385)
jbrockmendel Jan 25, 2021
b471c29
PERF: Rolling.cov/corr (#39388)
mroeschke Jan 25, 2021
a5ecf22
⬆️ UPGRADE: Autoupdate pre-commit config (#39391)
github-actions[bot] Jan 25, 2021
7d9ad04
TST: tighten Decimal tests (#39381)
jbrockmendel Jan 25, 2021
56476d1
BUG: Slicing DatetimeIndex with strings containing microseconds raisi…
phofl Jan 25, 2021
1dafea5
PERF: DatetimeIndex.get_indexer with mismatched tz (#39332)
jbrockmendel Jan 25, 2021
8dbf1cc
DOC: remove references to dev.pandas.io (#39377)
ayushmankumar7 Jan 25, 2021
62c7dd6
DOC: NDFrame fillna method add use case (#39109)
aniaan Jan 25, 2021
799143d
ENH: PandasArray.value_counts (#39387)
jbrockmendel Jan 25, 2021
b58e2b8
BUG: Series(list_of_tuples, dtype=PandasDtype(object)) (#39357)
jbrockmendel Jan 25, 2021
779592d
BUG: Series.resample fails on NaT index (#39229)
rhshadrach Jan 26, 2021
c519389
TST: remove test_error (#39386)
jbrockmendel Jan 26, 2021
3f07983
ENH: implement EA.delete (#39405)
jbrockmendel Jan 26, 2021
421fb8d
REF: implement Index._find_common_type_compat (#39408)
jbrockmendel Jan 26, 2021
ce3e57b
BUG: re-render CSS with `styler.apply` and `applymap` non-cleared ` _…
attack68 Jan 26, 2021
f61b7c5
BUG: update tests (#39189)
nofarmish Jan 26, 2021
8c3b3b6
BUG: update after precommit (#39189)
nofarmish Jan 26, 2021
4e3bce2
BUG: update after precommit (#39189)
nofarmish Jan 26, 2021
558a585
BUG: fix case of a category value which isn't exists (#39189)
nofarmish Jan 22, 2021
adfe600
BUG: add UT to conver_value for this use case (#39189)
nofarmish Jan 23, 2021
63815c7
BUG: change style with pre-commit (#39189)
nofarmish Jan 23, 2021
74c687a
BUG: add a whatsnew record (#39189)
nofarmish Jan 23, 2021
3023fc0
BUG: check for tests (#39189)
nofarmish Jan 23, 2021
f917ba9
BUG: remove spaces (#39189)
nofarmish Jan 23, 2021
0abe192
BUG: remove whatsnew (#39189)
nofarmish Jan 23, 2021
1b959ee
BUG: remove tests(#39189)
nofarmish Jan 23, 2021
4de349f
BUG: add whats new (#39189)
nofarmish Jan 23, 2021
d7a3ef6
BUG: check tests (#39189)
nofarmish Jan 23, 2021
eb8cd5a
BUG: update tests (#39189)
nofarmish Jan 26, 2021
235d05e
BUG: update after precommit (#39189)
nofarmish Jan 26, 2021
73541ff
BUG: update after precommit (#39189)
nofarmish Jan 26, 2021
877ae9e
Merge remote-tracking branch 'origin/read-hdf-returns-unexpected-valu…
nofarmish Jan 26, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.3.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,7 @@ I/O
- Bug in :func:`read_csv` not switching ``true_values`` and ``false_values`` for nullable ``boolean`` dtype (:issue:`34655`)
- Bug in :func:`read_json` when ``orient="split"`` does not maintain numeric string index (:issue:`28556`)
- :meth:`read_sql` returned an empty generator if ``chunksize`` was no-zero and the query returned no results. Now returns a generator with a single empty dataframe (:issue:`34411`)
- Bug in :func:`read_hdf` returning unexpected records when filtering on categorical string columns using ``where`` parameter (:issue:`39189`)

Period
^^^^^^
Expand Down
8 changes: 3 additions & 5 deletions pandas/core/computation/pytables.py
Original file line number Diff line number Diff line change
Expand Up @@ -210,12 +210,10 @@ def stringify(value):
return TermValue(int(v), v, kind)
elif meta == "category":
metadata = extract_array(self.metadata, extract_numpy=True)
result = metadata.searchsorted(v, side="left")

# result returns 0 if v is first element or if v is not in metadata
# check that metadata contains v
if not result and v not in metadata:
if v not in metadata:
result = -1
else:
result = metadata.searchsorted(v, side="left")
return TermValue(result, result, "integer")
elif kind == "integer":
v = int(float(v))
Expand Down
Empty file.