[ML] Exclude nested fields in data frame analytics #71400
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, the destination index was sorted which meant it could not have `nested` fields. Since this has changed, `nested` fields may be present. These were handled incorrectly as the _explain API would report that they can be included in the analysis while that is not the case. This commit fixes this issue by detecting `nested` fields and children of those `nested` fields and excluding them from the analysis. A `nested` field may contain multiple inner fields. To avoid the noise in the API response, we collapse them into a single entry with the path to the top level nested field.
|
Pinging @elastic/ml-core (Team:ML) |
benwtrent
approved these changes
Apr 7, 2021
dimitris-athanasiou
added a commit
that referenced
this pull request
Apr 7, 2021
) Previously, the destination index was sorted which meant it could not have `nested` fields. Since this has changed, `nested` fields may be present. These were handled incorrectly as the _explain API would report that they can be included in the analysis while that is not the case. This commit fixes this issue by detecting `nested` fields and children of those `nested` fields and excluding them from the analysis. A `nested` field may contain multiple inner fields. To avoid the noise in the API response, we collapse them into a single entry with the path to the top level nested field. Backport of #71400
dimitris-athanasiou
added a commit
to dimitris-athanasiou/elasticsearch
that referenced
this pull request
Apr 15, 2021
This fixes a bug that was introduced in elastic#71400. The bug occurs when the _explain API is called for a data frame analytics job and there are nested fields that are also incompatible. In particular, we end up calling `iterator.remove()` twice which throws `IllegalStateException`. I also took the chance to move the nested field check first as I think it's more informative to explain a field is not included due to being nested than because it has an incompatible type in this case. The PR is marked as `non-issue` as this has not been released yet.
dimitris-athanasiou
added a commit
that referenced
this pull request
Apr 15, 2021
This fixes a bug that was introduced in #71400. The bug occurs when the _explain API is called for a data frame analytics job and there are nested fields that are also incompatible. In particular, we end up calling `iterator.remove()` twice which throws `IllegalStateException`. I also took the chance to move the nested field check first as I think it's more informative to explain a field is not included due to being nested than because it has an incompatible type in this case. The PR is marked as `non-issue` as this has not been released yet.
dimitris-athanasiou
added a commit
that referenced
this pull request
Apr 15, 2021
…#71740) This fixes a bug that was introduced in #71400. The bug occurs when the _explain API is called for a data frame analytics job and there are nested fields that are also incompatible. In particular, we end up calling `iterator.remove()` twice which throws `IllegalStateException`. I also took the chance to move the nested field check first as I think it's more informative to explain a field is not included due to being nested than because it has an incompatible type in this case. The PR is marked as `non-issue` as this has not been released yet. Backport of #71736
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, the destination index was sorted which meant it could
not have
nestedfields. Since this has changed,nestedfieldsmay be present. These were handled incorrectly as the _explain API
would report that they can be included in the analysis while
that is not the case.
This commit fixes this issue by detecting
nestedfields and childrenof those
nestedfields and excluding them from the analysis. Anestedfield may contain multiple inner fields. To avoid the noisein the API response, we collapse them into a single entry with the
path to the top level nested field.