Skip to content

Fix SnapshotShardStatus Reporting for Failed Shard #48556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 29, 2019
Merged

Fix SnapshotShardStatus Reporting for Failed Shard #48556

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 3 additions & 8 deletions server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@
import org.apache.lucene.store.IndexOutput;
import org.apache.lucene.store.RateLimiter;
import org.apache.lucene.util.SetOnce;
import org.elasticsearch.ExceptionsHelper;
import org.elasticsearch.action.ActionListener;
import org.elasticsearch.action.ActionRunnable;
import org.elasticsearch.action.StepListener;
Expand Down Expand Up @@ -1042,10 +1041,6 @@ public void snapshotShard(Store store, MapperService mapperService, SnapshotId s
ActionListener<String> listener) {
final ShardId shardId = store.shardId();
final long startTime = threadPool.absoluteTimeInMillis();
final ActionListener<String> snapshotDoneListener = ActionListener.wrap(listener::onResponse, e -> {
snapshotStatus.moveToFailed(threadPool.absoluteTimeInMillis(), ExceptionsHelper.stackTrace(e));
listener.onFailure(e instanceof IndexShardSnapshotFailedException ? e : new IndexShardSnapshotFailedException(shardId, e));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One fewer instanceof in the world ❤️

});
try {
final String generation = snapshotStatus.generation();
logger.debug("[{}] [{}] snapshot to [{}] [{}] ...", shardId, snapshotId, metadata.name(), generation);
Expand Down Expand Up @@ -1191,8 +1186,8 @@ public void snapshotShard(Store store, MapperService mapperService, SnapshotId s
}
}
snapshotStatus.moveToDone(threadPool.absoluteTimeInMillis(), indexGeneration);
snapshotDoneListener.onResponse(indexGeneration);
}, snapshotDoneListener::onFailure);
listener.onResponse(indexGeneration);
}, listener::onFailure);
if (indexIncrementalFileCount == 0) {
allFilesUploadedListener.onResponse(Collections.emptyList());
return;
Expand Down Expand Up @@ -1222,7 +1217,7 @@ public void snapshotShard(Store store, MapperService mapperService, SnapshotId s
}));
}
} catch (Exception e) {
snapshotDoneListener.onFailure(e);
listener.onFailure(e);
}
}

Expand Down
4 changes: 3 additions & 1 deletion server/src/main/java/org/elasticsearch/snapshots/SnapshotShardsService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,8 +298,10 @@ public void onResponse(String newGeneration) {

@Override
public void onFailure(Exception e) {
final String failure = ExceptionsHelper.stackTrace(e);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ew. Can we follow up with a change that keeps the exception as an exception rather than converting it to a String here and in a few other places? Looks nontrivial because BWC, of course.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jup, I'm happy to try :)

snapshotStatus.moveToFailed(threadPool.absoluteTimeInMillis(), failure);
logger.warn(() -> new ParameterizedMessage("[{}][{}] failed to snapshot shard", shardId, snapshot), e);
notifyFailedSnapshotShard(snapshot, shardId, ExceptionsHelper.stackTrace(e));
notifyFailedSnapshotShard(snapshot, shardId, failure);
}
});
}
Expand Down
6 changes: 6 additions & 0 deletions server/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1219,6 +1219,12 @@ public void testDataNodeRestartWithBusyMasterDuringSnapshot() throws Exception {
disruption.startDisrupting();
logger.info("--> restarting data node, which should cause primary shards to be failed");
internalCluster().restartNode(dataNode, InternalTestCluster.EMPTY_CALLBACK);

logger.info("--> wait for shard snapshots to show as failed");
assertBusy(() -> assertThat(
Copy link
Contributor

@DaveCTurner DaveCTurner Oct 27, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this change we would sometimes unblock the node and stop the disruption before the first shard failure. I think this change makes the test weaker. I'm guessing it's invalid to do this after disruption.stopDisrupting()? If so, can we for instance only do it sometimes (with a comment saying why we don't always do it)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this change we would sometimes unblock the node and stop the disruption before the first shard failure.

I'd argue that's a good thing :) <= The whole point of this test was to test this situation (failure on the data node before CS updates resume). The case where we stop disrupting before anything fails is probably practically impossible and even if it wasn't something that's covered in SnapshotResiliencyTests (where want want that kind of randomness because we can reproduce things) anyway.
WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced yet. Practically impossible is not impossible enough for me :)

Do you think that the failure in #48526 is also captured, rarely, by the SnapshotResiliencyTests? Can we make a more focussed test there?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The situation we're running into is perfectly covered by SnapshotResiliencyTests but those tests don't use the status API (which is the only thing that's functionally broken here) so they don't fail.
We could maybe add a test that involves the status APIs to SnapshotResiliencyTests to cover these things there as well if we want more randomness. I think that would be better than purposely making this test run into an, let's say almost impossible, situation at some point and not be reproducible anyway?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, yes, SnapshotResiliencyTests sounds like a better place for this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd still rather adjust this test to have some reproducible testing for the concrete bug here and then enhance the SnapshotResiliencyTests down the line. That's basically what has been the strategy for other tests as well: make the ITs deterministic to reproduce known issues and and keep the randomness in SnapshotResiliencyTests.
Everything covered by this test is covered by SnapshotResiliencyTests anyway and this test was just added for 6.x coverage, so I don't see us losing any coverage here by making this one more deterministic? :)

client().admin().cluster().prepareSnapshotStatus("test-repo").setSnapshots("test-snap").get().getSnapshots()
.get(0).getShardsStats().getFailedShards(), greaterThanOrEqualTo(1)), 60L, TimeUnit.SECONDS);

unblockNode("test-repo", dataNode);
disruption.stopDisrupting();
// check that snapshot completes
Expand Down