Closed
Description
Elasticsearch version (bin/elasticsearch --version): 6.2.4
Plugins installed: [discovery-ec2 6.2.4, repository-s3 6.2.4]
JVM version (java -version): 1.8.0_161
OS version (uname -a if on a Unix-like system): Linux ip-10-4-0-82 4.4.0-1052-aws #61-Ubuntu SMP Mon Feb 12 23:05:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
9 days ago, I deleted a snapshot from an S3 repository. Elasticsearch logged that the snapshot was successfully deleted. When I try to get a list of snapshots from this repository that would include the deleted snapshot, I get a snapshot_missing_exception. When I try to delete this snapshot again, I get an error message notifying me that another snapshot is currently being deleted. (As far as I can tell, no other snapshot in this repository is currently being deleted.)
Steps to reproduce:
Unable to reproduce intentionally.
Provide logs (if relevant):
[2018-07-16T00:19:26,752][INFO ][o.e.s.SnapshotsService ] [prod-es-master-0bcee31972c150cea] snapshot [d3ac96df71f141df956b0fddc8618b2e_backup:d3ac96df71f141df956b0fddc8618b2e-2018-07-13__18787545/2eEFokOTTKuHFEjJQ8K85g] deleted
[2018-07-25T00:57:32,749][WARN ][r.suppressed ] path: /_snapshot/d3ac96df71f141df956b0fddc8618b2e_backup/d3ac96df71f141df956b0fddc8618b2e-2018-07-13__18787545, params: {pretty=, repository=d3ac96df71f141df956b0fddc8618b2e_backup, snapshot=d3ac96df71f141df956b0fddc8618b2e-2018-07-13__18787545}
org.elasticsearch.transport.RemoteTransportException: [prod-es-master-0bcee31972c150cea][10.4.1.79:9300][cluster:admin/snapshot/delete]
Caused by: org.elasticsearch.snapshots.ConcurrentSnapshotExecutionException: [d3ac96df71f141df956b0fddc8618b2e_backup:d3ac96df71f141df956b0fddc8618b2e-2018-07-13__18787545/2eEFokOTTKuHFEjJQ8K85g] cannot delete - another snapshot is currently being deleted
at org.elasticsearch.snapshots.SnapshotsService$6.execute(SnapshotsService.java:1121) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:273) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
[2018-07-25T01:40:59,466][WARN ][r.suppressed ] path: _snapshot/d3ac96df71f141df956b0fddc8618b2e_backup/_all, params: {repository=d3ac96df71f141df956b0fddc8618b2e_backup, snapshot=_all}
org.elasticsearch.transport.RemoteTransportException: [prod-es-master-0bcee31972c150cea][10.4.1.79:9300][cluster:admin/snapshot/get]
Caused by: org.elasticsearch.snapshots.SnapshotException: [d3ac96df71f141df956b0fddc8618b2e_backup:d3ac96df71f141df956b0fddc8618b2e-2018-07-13__18787545/2eEFokOTTKuHFEjJQ8K85g] Snapshot could not be read
at org.elasticsearch.snapshots.SnapshotsService.snapshots(SnapshotsService.java:197) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:136) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:55) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.elasticsearch.snapshots.SnapshotMissingException: [d3ac96df71f141df956b0fddc8618b2e_backup:d3ac96df71f141df956b0fddc8618b2e-2018-07-13__18787545/2eEFokOTTKuHFEjJQ8K85g] is missing
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getSnapshotInfo(BlobStoreRepository.java:493) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.snapshots.SnapshotsService.snapshots(SnapshotsService.java:191) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:136) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:55) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_161]
Caused by: java.nio.file.NoSuchFileException: Blob object [snap-2eEFokOTTKuHFEjJQ8K85g.dat] not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 904A7EC118E731BD; S3 Extended Request ID: iXkuSkq30rKoAgAWzfvsW/sSiUSGrQ2IOx2LgqQ0DeqFF8xLXyvp4h3rsYiToYu9tMtgGHRSbpA=)
at org.elasticsearch.repositories.s3.S3BlobContainer.readBlob(S3BlobContainer.java:90) ~[?:?]
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.readBlob(ChecksumBlobStoreFormat.java:103) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.repositories.blobstore.BlobStoreFormat.read(BlobStoreFormat.java:89) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getSnapshotInfo(BlobStoreRepository.java:491) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.snapshots.SnapshotsService.snapshots(SnapshotsService.java:191) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:136) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:55) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_161]