Wait for ML templates after creating a new cluster in TooManyJobsIT #54793
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/ml-core (:ml) |
droberts195
approved these changes
Apr 6, 2020
There was a problem hiding this comment.
LGTM
There's scope for changing the way we start these clusters more radically. The comment // clear all nodes, so that we can set xpack.ml.max_open_jobs setting around line 202/203 is wrong now - it dates back to 5.x. In 7.x xpack.ml.max_open_jobs is a dynamic cluster-wide setting, so we could change the setting with a single settings call rather than restarting the entire cluster. But I am happy to leave that for a future PR.
Yes I looked at that but I can't see how to change a setting on the internal cluster. There probably is a way I just couldn't see it |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TooManyJobsIT.testCloseFailedJobstops all the data nodes in the cluster and restarts them again, in effect this is a brand new cluster and the test should wait for the ml templates to be installed before continuing.This handles 2 of the failures listed in #54162 where indices in the new cluster are created without a template but there is still one failure where the test timed out waiting for the templates in the original cluster. This may just be due to a slow machine I will unmute the test in the backport to 7.x and watch for new failures.
As to why the tests staring failing now and only on 7.x I am not sure, possibly something in #51765. The best way forward is to unmute the test and see if it happens again