[CI] XPackRestIT failing on various YAML tests #75069
Assignees
Labels
:Data Management/Indices APIs
APIs to create and manage indices and templates
low-risk
An open issue or test failure that is a low risk to future releases
:ml
Machine learning
Team:Data Management
Meta label for data/management team
Team:ML
Meta label for the ML team
>test-failure
Triaged test failures from CI
Comments
romseygeek
added
:ml
|
Pinging @elastic/ml-core (Team:ML) |
romseygeek
added
the
:Data Management/Indices APIs
|
Pinging @elastic/es-core-features (Team:Core/Features) |
|
I think #73734 would fix this. As an interim fix we should probably just ignore this warning. |
droberts195
added a commit
to droberts195/elasticsearch
that referenced
this issue
Jul 8, 2021
This is a followup to elastic#74976. The changes of elastic#74976 reverted many of the changes of elastic#71656 because elastic#74415 made them redundant. elastic#74415 did this by making killed jobs as closing so that the standard "job closed immediately after open" functionality was used instead of reissuing the kill immediately after opening. However, it turns out that this "job closed immediately after open" functionality is not perfect for the case of a job that is killed while it is opening. It causes AutodetectCommunicator.close() to be called instead of AutodetectCommunicator.killProcess(). Both do a lot of the same things, but AutodetectCommunicator.close() finalizes the job, and this can cause problems if the job is being killed as part of a feature reset. This change reinstates some of the functionality of elastic#71656 but in a different place that hopefully won't reintroduce the problems that led to elastic#74415. We can detect that a kill has happened early on during an open or close operation by checking if the task's allocation ID has been removed from the map after ProcessContext.setDying() returns true. If ProcessContext.setDying() returns true this means the job has not been previously closed, so it must have been killed. Then we can call AutodetectCommunicator.killProcess() instead of AutodetectCommunicator.close() during the cleanup that happens when we detect that a recently started process is no longer wanted. Relates elastic#75069
|
A related failure in docs yaml tests: |
|
I think this is an evolution of #74141. After fixing some of the race conditions between ML job open, close and kill the races move on to code that runs later. In the failure from this issue's description the relevant bit of the server side log is this: Or, in plain English:
#75113 should fix this by changing the way we shut down the process in step 7 so that it should be free of side-effects. |
elasticsearchmachine
pushed a commit
that referenced
this issue
Jul 8, 2021
This is a followup to #74976. The changes of #74976 reverted many of the changes of #71656 because #74415 made them redundant. #74415 did this by making killed jobs as closing so that the standard "job closed immediately after open" functionality was used instead of reissuing the kill immediately after opening. However, it turns out that this "job closed immediately after open" functionality is not perfect for the case of a job that is killed while it is opening. It causes AutodetectCommunicator.close() to be called instead of AutodetectCommunicator.killProcess(). Both do a lot of the same things, but AutodetectCommunicator.close() finalizes the job, and this can cause problems if the job is being killed as part of a feature reset. This change reinstates some of the functionality of #71656 but in a different place that hopefully won't reintroduce the problems that led to #74415. We can detect that a kill has happened early on during an open or close operation by checking if the task's allocation ID has been removed from the map after ProcessContext.setDying() returns true. If ProcessContext.setDying() returns true this means the job has not been previously closed, so it must have been killed. Then we can call AutodetectCommunicator.killProcess() instead of AutodetectCommunicator.close() during the cleanup that happens when we detect that a recently started process is no longer wanted. Relates #75069
elasticsearchmachine
pushed a commit
to elasticsearchmachine/elasticsearch
that referenced
this issue
Jul 8, 2021
This is a followup to elastic#74976. The changes of elastic#74976 reverted many of the changes of elastic#71656 because elastic#74415 made them redundant. elastic#74415 did this by making killed jobs as closing so that the standard "job closed immediately after open" functionality was used instead of reissuing the kill immediately after opening. However, it turns out that this "job closed immediately after open" functionality is not perfect for the case of a job that is killed while it is opening. It causes AutodetectCommunicator.close() to be called instead of AutodetectCommunicator.killProcess(). Both do a lot of the same things, but AutodetectCommunicator.close() finalizes the job, and this can cause problems if the job is being killed as part of a feature reset. This change reinstates some of the functionality of elastic#71656 but in a different place that hopefully won't reintroduce the problems that led to elastic#74415. We can detect that a kill has happened early on during an open or close operation by checking if the task's allocation ID has been removed from the map after ProcessContext.setDying() returns true. If ProcessContext.setDying() returns true this means the job has not been previously closed, so it must have been killed. Then we can call AutodetectCommunicator.killProcess() instead of AutodetectCommunicator.close() during the cleanup that happens when we detect that a recently started process is no longer wanted. Relates elastic#75069
elasticsearchmachine
pushed a commit
to elasticsearchmachine/elasticsearch
that referenced
this issue
Jul 8, 2021
This is a followup to elastic#74976. The changes of elastic#74976 reverted many of the changes of elastic#71656 because elastic#74415 made them redundant. elastic#74415 did this by making killed jobs as closing so that the standard "job closed immediately after open" functionality was used instead of reissuing the kill immediately after opening. However, it turns out that this "job closed immediately after open" functionality is not perfect for the case of a job that is killed while it is opening. It causes AutodetectCommunicator.close() to be called instead of AutodetectCommunicator.killProcess(). Both do a lot of the same things, but AutodetectCommunicator.close() finalizes the job, and this can cause problems if the job is being killed as part of a feature reset. This change reinstates some of the functionality of elastic#71656 but in a different place that hopefully won't reintroduce the problems that led to elastic#74415. We can detect that a kill has happened early on during an open or close operation by checking if the task's allocation ID has been removed from the map after ProcessContext.setDying() returns true. If ProcessContext.setDying() returns true this means the job has not been previously closed, so it must have been killed. Then we can call AutodetectCommunicator.killProcess() instead of AutodetectCommunicator.close() during the cleanup that happens when we detect that a recently started process is no longer wanted. Relates elastic#75069
elasticsearchmachine
added a commit
that referenced
this issue
Jul 8, 2021
…5116) This is a followup to #74976. The changes of #74976 reverted many of the changes of #71656 because #74415 made them redundant. #74415 did this by making killed jobs as closing so that the standard "job closed immediately after open" functionality was used instead of reissuing the kill immediately after opening. However, it turns out that this "job closed immediately after open" functionality is not perfect for the case of a job that is killed while it is opening. It causes AutodetectCommunicator.close() to be called instead of AutodetectCommunicator.killProcess(). Both do a lot of the same things, but AutodetectCommunicator.close() finalizes the job, and this can cause problems if the job is being killed as part of a feature reset. This change reinstates some of the functionality of #71656 but in a different place that hopefully won't reintroduce the problems that led to #74415. We can detect that a kill has happened early on during an open or close operation by checking if the task's allocation ID has been removed from the map after ProcessContext.setDying() returns true. If ProcessContext.setDying() returns true this means the job has not been previously closed, so it must have been killed. Then we can call AutodetectCommunicator.killProcess() instead of AutodetectCommunicator.close() during the cleanup that happens when we detect that a recently started process is no longer wanted. Relates #75069 Co-authored-by: David Roberts <[email protected]>
elasticsearchmachine
added a commit
that referenced
this issue
Jul 8, 2021
…5117) This is a followup to #74976. The changes of #74976 reverted many of the changes of #71656 because #74415 made them redundant. #74415 did this by making killed jobs as closing so that the standard "job closed immediately after open" functionality was used instead of reissuing the kill immediately after opening. However, it turns out that this "job closed immediately after open" functionality is not perfect for the case of a job that is killed while it is opening. It causes AutodetectCommunicator.close() to be called instead of AutodetectCommunicator.killProcess(). Both do a lot of the same things, but AutodetectCommunicator.close() finalizes the job, and this can cause problems if the job is being killed as part of a feature reset. This change reinstates some of the functionality of #71656 but in a different place that hopefully won't reintroduce the problems that led to #74415. We can detect that a kill has happened early on during an open or close operation by checking if the task's allocation ID has been removed from the map after ProcessContext.setDying() returns true. If ProcessContext.setDying() returns true this means the job has not been previously closed, so it must have been killed. Then we can call AutodetectCommunicator.killProcess() instead of AutodetectCommunicator.close() during the cleanup that happens when we detect that a recently started process is no longer wanted. Relates #75069 Co-authored-by: David Roberts <[email protected]>
|
Another one just happened in a docs test: https://gradle-enterprise.elastic.co/s/q6lylbs4wg3we However, this one is for the The relevant bit of the server-side log is: So it looks like the |
|
This is another one that failed due to In this case the test that failed was @williamrandolph do you think you would be able to merge #73734 soon with all the nice-to-haves deferred to a followup? I think that would be good enough for the docs tests. The only other solution would be to disable snippet testing for all the snippets added in #74975 that use async search. /cc @jrodewig |
|
I see a lot of failures in yaml tests this this "unexpected warning header" today, its one of the biggest sources of failures at the moment (although the overall frequency is bareable), so I'm +1 on gettin #73734 in as soon as @williamrandolph thinks its possible to get the noise level down. |
elasticsearchmachine
pushed a commit
that referenced
this issue
Jul 12, 2021
elasticsearchmachine
pushed a commit
to elasticsearchmachine/elasticsearch
that referenced
this issue
Jul 12, 2021
This was referenced Jul 12, 2021
elasticsearchmachine
pushed a commit
to elasticsearchmachine/elasticsearch
that referenced
this issue
Jul 12, 2021
elasticsearchmachine
added a commit
that referenced
this issue
Jul 12, 2021
Co-authored-by: James Rodewig <[email protected]>
elasticsearchmachine
added a commit
that referenced
this issue
Jul 12, 2021
Co-authored-by: James Rodewig <[email protected]>
|
@droberts195 Thanks for the ping. I will bring PR #73734 up to date and merge it as soon as I can. |
|
@williamrandolph are you still planning to merge #73734 to address this? |
|
Yes, now that we've finished our features for 8.0, I need to take another pass at that PR. Thanks for the reminder. |
droberts195
added
the
low-risk
|
Pinging @elastic/es-data-management (Team:Data Management) |
joegallo
added a commit
to joegallo/elasticsearch
that referenced
this issue
Mar 8, 2024
This reverts commit 75c585c.
|
I talked to @williamrandolph and between #94130 and #94191 we think this one's probably closable. I put up #106140 to revert #75237, and once that's merged I'm going to close this ticket. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Build scan:
https://gradle-enterprise.elastic.co/s/svd6gy2xzpfn4/tests/:x-pack:plugin:yamlRestTest/org.elasticsearch.xpack.test.rest.XPackRestIT/test%20%7Bp0=data_stream%2F140_data_stream_aliases%2FCreate%20data%20stream%20alias%7D
Reproduction line:
./gradlew ':x-pack:plugin:yamlRestTest' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=data_stream/140_data_stream_aliases/Create data stream alias}" -Dtests.seed=9F9D630801B00349 -Dtests.locale=it-IT -Dtests.timezone=Asia/Ashkhabad -Druntime.java=8Applicable branches:
7.x, master
Reproduces locally?:
No
Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.test.rest.XPackRestIT&tests.test=test%20%7Bp0%3Ddata_stream/140_data_stream_aliases/Create%20data%20stream%20alias%7D
Failure excerpt:
The text was updated successfully, but these errors were encountered: