Skip to content

[CI] Test cat data frame analytics single job with header failing #58841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ywelsch opened this issue Jul 1, 2020 · 4 comments · Fixed by #59027 or #61180
Closed

[CI] Test cat data frame analytics single job with header failing #58841

ywelsch opened this issue Jul 1, 2020 · 4 comments · Fixed by #59027 or #61180
Labels
:ml Machine learning >test-failure Triaged test failures from CI

Comments

@ywelsch
Copy link
Contributor

ywelsch commented Jul 1, 2020

Failed 3 times on 7.x in the past days

Build scan: https://gradle-enterprise.elastic.co/s/7vn6gnjkkthfy

Output:

org.elasticsearch.xpack.test.rest.XPackRestIT > test {p0=ml/data_frame_analytics_cat_apis/Test cat data frame analytics single job with header} FAILED
    org.elasticsearch.client.ResponseException: method [DELETE], host [http://127.0.0.1:35015], URI [/_ml/data_frame/analytics/dfa-classification-job], status line [HTTP/1.1 503 Service Unavailable]
    {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
        at __randomizedtesting.SeedInfo.seed([22DBE7B704A1086E:AA8FD86DAA5D6596]:0)
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:293)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:263)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:269)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:237)
        at org.elasticsearch.xpack.core.ml.integration.MlRestTestStateCleaner.deleteAllDataFrameAnalytics(MlRestTestStateCleaner.java:110)
        at org.elasticsearch.xpack.core.ml.integration.MlRestTestStateCleaner.clearMlMetadata(MlRestTestStateCleaner.java:32)
        at org.elasticsearch.xpack.test.rest.XPackRestIT.clearMlState(XPackRestIT.java:222)
        at org.elasticsearch.xpack.test.rest.XPackRestIT.cleanup(XPackRestIT.java:206)
@ywelsch ywelsch added >test-failure Triaged test failures from CI :ml/Transform Transform labels Jul 1, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml/Transform)

@hendrikmuhs hendrikmuhs added :ml Machine learning and removed :ml/Transform Transform labels Jul 3, 2020
@droberts195 droberts195 mentioned this issue Jul 3, 2020
Closed
droberts195 added a commit to droberts195/elasticsearch that referenced this issue Jul 3, 2020
@droberts195
[ML] Wait for shards to initialize after creating ML internal indices
6758c96 
There have been a few test failures that are likely caused by tests
performing actions that use ML indices immediately after the actions
that create those ML indices.  Currently this can result in attempts
to search the newly created index before its shards have initialized.

This change makes the method that creates the internal ML indices
that have been affected by this problem (state and stats) wait for
the shards to be initialized before returning.

Fixes elastic#54887
Fixes elastic#55221
Fixes elastic#55807
Fixes elastic#57102
Fixes elastic#58841
Fixes elastic#59011
@droberts195 droberts195 mentioned this issue Jul 3, 2020
Merged
droberts195 added a commit that referenced this issue Jul 6, 2020
@droberts195
[ML] Wait for shards to initialize after creating ML internal indices (…
0804b8c 
…#59027)

There have been a few test failures that are likely caused by tests
performing actions that use ML indices immediately after the actions
that create those ML indices.  Currently this can result in attempts
to search the newly created index before its shards have initialized.

This change makes the method that creates the internal ML indices
that have been affected by this problem (state and stats) wait for
the shards to be initialized before returning.

Fixes #54887
Fixes #55221
Fixes #55807
Fixes #57102
Fixes #58841
Fixes #59011
@cbuescher
Copy link
Member

This looks similar to the issue mentioned here. @droberts195 should this be fixed on 7.x or is this a different issue?
https://gradle-enterprise.elastic.co/s/xnrdv7s6r64ks

@cbuescher cbuescher reopened this Jul 21, 2020
@droberts195
Copy link
Contributor

Another failure in https://gradle-enterprise.elastic.co/s/rbuxasr7s4yug

It's most likely the TrainedModelStatsService asynchronously creating indices part way through other tests. In #60462 (comment) the problem occurred during cleanup. In this test failure it's occurring part way through the test. The _cat API doesn't mind the stats index don't exist at all, as they won't if none of the data frame analytics models have ever been started. But it gets tripped up if the stats index exists but isn't searchable.

@przemekwitek
Copy link
Contributor

Happened again in https://gradle-enterprise.elastic.co/s/wmrnvevbdno7i

@davidkyle davidkyle mentioned this issue Aug 17, 2020
Merged
@benwtrent benwtrent mentioned this issue Dec 3, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:ml Machine learning >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
6 participants
@ywelsch @hendrikmuhs @droberts195 @cbuescher @elasticmachine @przemekwitek