add exclude_keys option to KeyValueProcessor
#24876
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
talevy
added
:Data Management/Ingest Node
jpountz
requested changes
May 30, 2017
| .filter((p) -> includeKeys == null || includeKeys.contains(p[0])) | ||
| .filter((p) -> | ||
| (includeKeys == null || includeKeys.contains(p[0])) && | ||
| (excludeKeys == null || excludeKeys.contains(p[0]) == false)) |
There was a problem hiding this comment.
Those contains calls have a linear complexity due to the fact we are using lists, let's use hash sets instead? (and maybe even automata as a follow-up?)
There was a problem hiding this comment.
thanks, I suppose I assumed the cost is minimal since this list is not expected to be more than just a few items... but that is being multiplied by millions of documents, so good call
updated
and modify data-structure of `include_keys` and `exclude_keys` to be backed by a HashSet
|
@jpountz thanks for the review! what do you think about the follow-up changes? |
talevy
added a commit
that referenced
this pull request
Jun 5, 2017
and modify data-structure of `include_keys` and `exclude_keys` to be backed by a HashSet
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Jun 6, 2017
* master: Add support for clear scroll to high level REST client (elastic#25038) Tiny correction in inner-hits.asciidoc (elastic#25066) Added release notes for 6.0.0-alpha2 Expand index expressions against indices only when managing aliases (elastic#23997) Collapse inner hits rest test should not skip 5.x Settings: Fix secure settings by prefix (elastic#25064) add `exclude_keys` option to KeyValueProcessor (elastic#24876) Test: update missing body tests to run against versions >= 5.5.0 Track EWMA[1] of task execution time in search threadpool executor Removes an invalid assert in resizing big arrays which does not always hold (resizing can result in a smaller size than the current size, while the assert attempted to verify the new size is always greater than the current). Fixed NPEs caused by requests without content. (elastic#23497) Plugins can register pre-configured char filters (elastic#25000) Build: Allow preserving shared dir (elastic#24962) Tests: Make secure settings available from settings builder for tests (elastic#25037)
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Jun 6, 2017
* master: (1210 commits) Add support for clear scroll to high level REST client (elastic#25038) Tiny correction in inner-hits.asciidoc (elastic#25066) Added release notes for 6.0.0-alpha2 Expand index expressions against indices only when managing aliases (elastic#23997) Collapse inner hits rest test should not skip 5.x Settings: Fix secure settings by prefix (elastic#25064) add `exclude_keys` option to KeyValueProcessor (elastic#24876) Test: update missing body tests to run against versions >= 5.5.0 Track EWMA[1] of task execution time in search threadpool executor Removes an invalid assert in resizing big arrays which does not always hold (resizing can result in a smaller size than the current size, while the assert attempted to verify the new size is always greater than the current). Fixed NPEs caused by requests without content. (elastic#23497) Plugins can register pre-configured char filters (elastic#25000) Build: Allow preserving shared dir (elastic#24962) Tests: Make secure settings available from settings builder for tests (elastic#25037) [TEST] Skip wildcard expansion test due to breaking change Test that gradle and Java version types match (elastic#24943) Include duplicate jar when jarhell check fails Change ScriptContexts to use needs instead of uses$. (elastic#25036) Change `has_child`, `has_parent` queries and `childen` aggregation to work with the new join field type and at the same time maintaining support for the `_parent` meta field type. Remove comma-separated feature parsing for GetIndicesAction ...
jasontedor
added a commit
to zeha/elasticsearch
that referenced
this pull request
Jun 6, 2017
* master: (619 commits) Add support for clear scroll to high level REST client (elastic#25038) Tiny correction in inner-hits.asciidoc (elastic#25066) Added release notes for 6.0.0-alpha2 Expand index expressions against indices only when managing aliases (elastic#23997) Collapse inner hits rest test should not skip 5.x Settings: Fix secure settings by prefix (elastic#25064) add `exclude_keys` option to KeyValueProcessor (elastic#24876) Test: update missing body tests to run against versions >= 5.5.0 Track EWMA[1] of task execution time in search threadpool executor Removes an invalid assert in resizing big arrays which does not always hold (resizing can result in a smaller size than the current size, while the assert attempted to verify the new size is always greater than the current). Fixed NPEs caused by requests without content. (elastic#23497) Plugins can register pre-configured char filters (elastic#25000) Build: Allow preserving shared dir (elastic#24962) Tests: Make secure settings available from settings builder for tests (elastic#25037) [TEST] Skip wildcard expansion test due to breaking change Test that gradle and Java version types match (elastic#24943) Include duplicate jar when jarhell check fails Change ScriptContexts to use needs instead of uses$. (elastic#25036) Change `has_child`, `has_parent` queries and `childen` aggregation to work with the new join field type and at the same time maintaining support for the `_parent` meta field type. Remove comma-separated feature parsing for GetIndicesAction ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
adds
exclude_keysoption, just like Logstash allows. Where, if included, a field will not show up in the resulting document. Attempts to preserve exact behavior.Closes #23856