Skip to content

FeaturesIT » testResetFeatures #71072

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
benwtrent opened this issue Mar 30, 2021 · 1 comment · Fixed by #71081
Closed

FeaturesIT » testResetFeatures #71072

benwtrent opened this issue Mar 30, 2021 · 1 comment · Fixed by #71081
Assignees
@benwtrent
Labels
:ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@benwtrent
Copy link
Member

benwtrent commented Mar 30, 2021 •
edited
Loading

Build scan:
https://gradle-enterprise.elastic.co/s/wvcoydmkr7mf4
Repro line:
./gradlew ':client:rest-high-level:asyncIntegTest' --tests "org.elasticsearch.client.FeaturesIT.testResetFeatures" -Dtests.seed=DC0F6E0DE2734BED -Dtests.security.manager=true -Dtests.locale=uk -Dtests.timezone=Etc/GMT-8 -Druntime.java=11
Reproduces locally?:
eventually
Applicable branches:
master

Failure excerpt:

org.elasticsearch.client.FeaturesIT > testResetFeatures FAILED
    org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=runtime_exception, reason=Unable to reset machine learning feature as there are ingest pipelines still referencing trained machine learning models]
        at __randomizedtesting.SeedInfo.seed([DC0F6E0DE2734BED:8F8A2A76A94F6C54]:0)
        at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176)
        at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1933)
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1910)
        at org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:1826)
        at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:648)
        at org.elasticsearch.client.RestClient$1.completed(RestClient.java:390)
        at org.elasticsearch.client.RestClient$1.completed(RestClient.java:374)
        at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.base/java.lang.Thread.run(Thread.java:834)

So, the ML tests didn't clean up their trained models or pipelines 🤦 fixing that
@benwtrent benwtrent added >test-failure Triaged test failures from CI :ml Machine learning labels Mar 30, 2021
@benwtrent benwtrent self-assigned this Mar 30, 2021
@elasticmachine elasticmachine added the Team:ML Meta label for the ML team label Mar 30, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

benwtrent added a commit to benwtrent/elasticsearch that referenced this issue Mar 30, 2021
@benwtrent
[ML] test mute for issue elastic#71072
bf1523a
benwtrent added a commit that referenced this issue Mar 30, 2021
@benwtrent
[ML] test mute for issue #71072 (#71073)
2450773
benwtrent added a commit to benwtrent/elasticsearch that referenced this issue Mar 30, 2021
@benwtrent
Test mute for ml reset change elastic#71072
c832446
benwtrent added a commit that referenced this issue Mar 30, 2021
@benwtrent
Test mute for ml reset change #71072 (#71075)
e78cf3c
@benwtrent benwtrent mentioned this issue Mar 30, 2021
Merged
benwtrent added a commit that referenced this issue Mar 31, 2021
@benwtrent
[ML] fixing feature reset integration tests (#71081)
ec9d062 
previously created pipelines referencing ML models were not being appropriately deleted in upstream tests.

This commit ensures that machine learning removes relevant pipelines from cluster state after tests complete

closes #71072
benwtrent added a commit to benwtrent/elasticsearch that referenced this issue Mar 31, 2021
@benwtrent
[ML] fixing feature reset integration tests (elastic#71081)
bde4f40 
previously created pipelines referencing ML models were not being appropriately deleted in upstream tests.

This commit ensures that machine learning removes relevant pipelines from cluster state after tests complete

closes elastic#71072
benwtrent added a commit that referenced this issue Mar 31, 2021
@benwtrent
[7.x] [ML] complete machine learning plugin feature state clean up in…
4260f1f 
…tegration (#71011) (#71071)

* [ML] complete machine learning plugin feature state clean up integration (#71011)

This completes the machine learning feature state cleanup integration.

This commit handles waiting for machine learning tasks to complete and adds a new
field to the ML Metadata cluster state to indicate when a reset is in progress for machine
learning.

relates: #70008

* [ML] fixing feature reset integration tests (#71081)

previously created pipelines referencing ML models were not being appropriately deleted in upstream tests.

This commit ensures that machine learning removes relevant pipelines from cluster state after tests complete

closes #71072
@ldematte ldematte mentioned this issue Mar 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benwtrent benwtrent

Labels
:ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML] fixing feature reset integration tests benwtrent/elasticsearch
2 participants
@benwtrent @elasticmachine