Skip to content

CI Failures for SLMSnapshotBlockingIntegTests testSnapshotInProgress #47689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jdconrad opened this issue Oct 7, 2019 · 5 comments · Fixed by #48020
Closed

CI Failures for SLMSnapshotBlockingIntegTests testSnapshotInProgress #47689

jdconrad opened this issue Oct 7, 2019 · 5 comments · Fixed by #48020
Assignees
@original-brownbear
Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >test-failure Triaged test failures from CI v8.0.0-alpha1

Comments

@jdconrad
Copy link
Contributor

jdconrad commented Oct 7, 2019 •
edited by dakrone
Loading

Reproduce with:

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ilm:test' --tests "org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests.testSnapshotInProgress" \
  -Dtests.seed=65DEEE302AEB2E47 \
  -Dtests.security.manager=true \
  -Dtests.locale=hsb-DE \
  -Dtests.timezone=America/Indiana/Petersburg \
  -Dcompiler.java=12 \
  -Druntime.java=12

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ilm:test' --tests "org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests.testSnapshotInProgress" \
  -Dtests.seed=65DEEE302AEB2E47 \
  -Dtests.security.manager=true \
  -Dtests.locale=hsb-DE \
  -Dtests.timezone=America/Indiana/Petersburg \
  -Dcompiler.java=12 \
  -Druntime.java=12

from (https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=openjdk12,nodes=general-purpose/251/console)

https://gradle-enterprise.elastic.co/s/cc7dkdd6bvbfs
@jdconrad jdconrad added >test-failure Triaged test failures from CI :Data Management/ILM+SLM Index and Snapshot lifecycle management v8.0.0 labels Oct 7, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

@jimczi jimczi mentioned this issue Oct 9, 2019
Merged
jimczi added a commit that referenced this issue Oct 9, 2019
@jimczi
Disable SLMSnapshotBlockingIntegTests#testSnapshotInProgress (#47775)
1486137 
This test fails constantly in master and prs.

Relates #47689
jimczi added a commit that referenced this issue Oct 9, 2019
@jimczi
Disable SLMSnapshotBlockingIntegTests#testSnapshotInProgress (#47775)
d969772 
This test fails constantly in master and prs.

Relates #47689
@jimczi
Copy link
Contributor

jimczi commented Oct 9, 2019

The test is now muted in master and 7x.

@original-brownbear original-brownbear self-assigned this Oct 10, 2019
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Oct 14, 2019
@original-brownbear
Safely Close Repositories on Node Shutdown
017d35c 
We were not closing repositories on Node shutdown.
In production, this has little effect but in tests
shutting down a node using `MockRepository` and is
currently stuck in a simulated blocked-IO situation
will only unblock when the node's threadpool is
interrupted. This might in some edge cases (many
snapshot threads and some CI slowness) result
in the execution taking longer than 5s to release
all the shard stores and thus we fail the assertion
about unreleased shard stores in the internal test cluster.

Regardless of tests, I think we should close repositories
and release resources associated with them when closing
a node and not just when removing a repository from the CS
with running nodes as this behavior is really unexpected.

Fixes elastic#47689
@original-brownbear original-brownbear mentioned this issue Oct 14, 2019
Merged
original-brownbear added a commit that referenced this issue Oct 16, 2019
@original-brownbear
Safely Close Repositories on Node Shutdown (#48020)
27c24e2 
We were not closing repositories on Node shutdown.
In production, this has little effect but in tests
shutting down a node using `MockRepository` and is
currently stuck in a simulated blocked-IO situation
will only unblock when the node's threadpool is
interrupted. This might in some edge cases (many
snapshot threads and some CI slowness) result
in the execution taking longer than 5s to release
all the shard stores and thus we fail the assertion
about unreleased shard stores in the internal test cluster.

Regardless of tests, I think we should close repositories
and release resources associated with them when closing
a node and not just when removing a repository from the CS
with running nodes as this behavior is really unexpected.

Fixes #47689
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Oct 16, 2019
@original-brownbear
Safely Close Repositories on Node Shutdown (elastic#48020)
b80e982 
We were not closing repositories on Node shutdown.
In production, this has little effect but in tests
shutting down a node using `MockRepository` and is
currently stuck in a simulated blocked-IO situation
will only unblock when the node's threadpool is
interrupted. This might in some edge cases (many
snapshot threads and some CI slowness) result
in the execution taking longer than 5s to release
all the shard stores and thus we fail the assertion
about unreleased shard stores in the internal test cluster.

Regardless of tests, I think we should close repositories
and release resources associated with them when closing
a node and not just when removing a repository from the CS
with running nodes as this behavior is really unexpected.

Fixes elastic#47689
@original-brownbear original-brownbear mentioned this issue Oct 16, 2019
Merged
original-brownbear added a commit that referenced this issue Oct 17, 2019
@original-brownbear
Safely Close Repositories on Node Shutdown (#48020) (#48107)
0ca7cc1 
We were not closing repositories on Node shutdown.
In production, this has little effect but in tests
shutting down a node using `MockRepository` and is
currently stuck in a simulated blocked-IO situation
will only unblock when the node's threadpool is
interrupted. This might in some edge cases (many
snapshot threads and some CI slowness) result
in the execution taking longer than 5s to release
all the shard stores and thus we fail the assertion
about unreleased shard stores in the internal test cluster.

Regardless of tests, I think we should close repositories
and release resources associated with them when closing
a node and not just when removing a repository from the CS
with running nodes as this behavior is really unexpected.

Fixes #47689
@williamrandolph
Copy link
Contributor

This test failed on one of my intake builds: https://gradle-enterprise.elastic.co/s/f7yigor4zvm2w/


org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests > testSnapshotInProgress FAILED
--
RepositoryException[[repo-id] failed to delete snapshot [snap-x38p3s0cqx20q6dbavuhoq/BeAykT7oT3us694vG0sKdA]]; nested: RepositoryException[[repo-id] could not read repository data from index blob]; nested: NoSuchFileException[/dev/shm/elastic+elasticsearch+7.x+multijob+fast+part2/x-pack/plugin/ilm/build/testrun/test/temp/org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests_349BD365F30B51E5-001/tempDir-004/repos/RTRmjH/index-0];
 
  Caused by:
    RepositoryException[[repo-id] could not read repository data from index blob]; nested: NoSuchFileException[/dev/shm/elastic+elasticsearch+7.x+multijob+fast+part2/x-pack/plugin/ilm/build/testrun/test/temp/org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests_349BD365F30B51E5-001/tempDir-004/repos/RTRmjH/index-0];
 
    Caused by:
      java.nio.file.NoSuchFileException: /dev/shm/elastic+elasticsearch+7.x+multijob+fast+part2/x-pack/plugin/ilm/build/testrun/test/temp/org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests_349BD365F30B51E5-001/tempDir-004/repos/RTRmjH/index-0

REPRODUCE WITH: ./gradlew ':x-pack:plugin:ilm:test' --tests "org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests.testSnapshotInProgress" -Dtests.seed=349BD365F30B51E5 -Dtests.security.manager=true -Dtests.locale=mt-MT -Dtests.timezone=Canada/Mountain -Dcompiler.java=12 -Druntime.java=8

I can't tell if this is the same issue we were seeing before or not.

@dakrone
Copy link
Member

dakrone commented Nov 15, 2019

I think Armin just fixed this in #49142, @original-brownbear is my understanding of that correct?

@original-brownbear
Copy link
Member

@dakrone jup just merged the backports to 7.x and 7.5 a few seconds ago :) We should be good here now.

@tlrx tlrx mentioned this issue Nov 19, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >test-failure Triaged test failures from CI v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Safely Close Repositories on Node Shutdown original-brownbear/elasticsearch
7 participants
@dakrone @jakelandis @jdconrad @williamrandolph @original-brownbear @elasticmachine @jimczi