Improve reduction of terms aggregations (#61779) #62028
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Improve reduction of terms aggregations Today, the terms aggregation reduces multiple aggregations at once using a map to group same buckets together. This operation can be costly since it requires to lookup every bucket in a global map with no particular order. This commit changes how term buckets are sorted by shards and partial reduces in order to be able to reduce results using a merge-sort strategy. For bwc, results are merged with the legacy code if any of the aggregations use a different sort (if it was returned by a node in prior versions). Relates elastic#51857
jimczi
added a commit
to jimczi/elasticsearch
that referenced
this pull request
Sep 7, 2020
This commit adapts the bwc version checks added in elastic#61779 and disable bwc tests until the backport elastic#62028 is merged.
jimczi
added a commit
that referenced
this pull request
Sep 7, 2020
jimczi
added a commit
to jimczi/elasticsearch
that referenced
this pull request
Sep 21, 2020
This commit ensures that the final order of the terms aggregations is registered correctly after the final reduce. This bug was introduced in elastic#62028 which is not released yet so this PR is marked as a non-issue. This issue was discovered when running a terms aggregation under an auto-date histogram. In such a case, the auto-date histogram may run multiple final reduce to merge buckets together. This change makes sure that running multiple final reduces doesn't create duplicates but it doesn't fix the fact that the final reduce may prune the list of terms prematurely. This other bug is tracked separately in elastic#62731.
jimczi
added a commit
that referenced
this pull request
Sep 21, 2020
This commit ensures that the final order of the terms aggregations is registered correctly after the final reduce. This bug was introduced in #62028 which is not released yet so this PR is marked as a non-issue. This issue was discovered when running a terms aggregation under an auto-date histogram. In such a case, the auto-date histogram may run multiple final reduce to merge buckets together. This change makes sure that running multiple final reduces doesn't create duplicates but it doesn't fix the fact that the final reduce may prune the list of terms prematurely. This other bug is tracked separately in #62731.
jimczi
added a commit
that referenced
this pull request
Sep 21, 2020
This commit ensures that the final order of the terms aggregations is registered correctly after the final reduce. This bug was introduced in #62028 which is not released yet so this PR is marked as a non-issue. This issue was discovered when running a terms aggregation under an auto-date histogram. In such a case, the auto-date histogram may run multiple final reduce to merge buckets together. This change makes sure that running multiple final reduces doesn't create duplicates but it doesn't fix the fact that the final reduce may prune the list of terms prematurely. This other bug is tracked separately in #62731.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of #61779