Skip to content

GCS SDK does not resume downloads that fail part-way through #52319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
DaveCTurner opened this issue Feb 13, 2020 · 1 comment · Fixed by #52479
Closed

GCS SDK does not resume downloads that fail part-way through #52319

DaveCTurner opened this issue Feb 13, 2020 · 1 comment · Fixed by #52479
Assignees
@original-brownbear
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement

Comments

@DaveCTurner
Copy link
Contributor

A connection dropped part-way through the download of a blob from GCS is not today considered retryable by the SDK, so a restore from GCS over a flaky connection may fail and need to be retried from the beginning.

I've seen two different (but related) exceptions:

[2020-02-13T12:10:57,150][WARN ][o.e.c.r.a.AllocationService] [elasticsearch-2] failing shard [failed shard, shard [elasticlogs-2018-05-13][0], node[d2sAR621RziQbILiZMBvfg], [P], recovery_source[snapshot recovery [iJoJDpgyR9GjOYO1enQmjQ] from small_logging_cluster:snapshot_80gb/fioJzVMyTcKPCRdfN5lZ7w], s[INITIALIZING], a[id=hi-CTSfzRliMbh3ajtzVVg], unassigned_info[[reason=NEW_INDEX_RESTORED], at[2020-02-13T11:22:30.460Z], delayed=false, details[restore_source[small_logging_cluster/snapshot_80gb]], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[elasticlogs-2018-05-13][0]: Recovery failed on {elasticsearch-1}{d2sAR621RziQbILiZMBvfg}{cgzSxj1tSjawYThZfSEXpQ}{10.10.9.72}{10.10.9.72:9300}{dim}{xpack.installed=true}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot_80gb/fioJzVMyTcKPCRdfN5lZ7w]]; nested: NotSerializableExceptionWrapper[storage_exception: Connection reset]; nested: IOException[Connection reset]; nested: IOException[Connection reset]; ], markAsStale [true]]
org.elasticsearch.indices.recovery.RecoveryFailedException: [elasticlogs-2018-05-13][0]: Recovery failed on {elasticsearch-1}{d2sAR621RziQbILiZMBvfg}{cgzSxj1tSjawYThZfSEXpQ}{10.10.9.72}{10.10.9.72:9300}{dim}{xpack.installed=true}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2584) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.1.jar:7.5.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:353) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:480) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [snapshot_80gb/fioJzVMyTcKPCRdfN5lZ7w]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:1255) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.Repository.restoreShard(Repository.java:231) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:469) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: storage_exception: Connection reset
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:227) ~[?:?]
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:690) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124) ~[?:?]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[?:?]
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:123) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.lambda$read$0(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
	at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.read(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:?]
	at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-7.5.1.jar:7.5.1]
	at java.io.InputStream.read(InputStream.java:213) ~[?:?]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFile(FileRestoreContext.java:240) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFiles(FileRestoreContext.java:217) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:180) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:1253) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.Repository.restoreShard(Repository.java:231) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:469) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
Caused by: java.io.IOException: Connection reset
	at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:137) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1463) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1431) ~[?:?]
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1022) ~[?:?]
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:245) ~[?:?]
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:285) ~[?:?]
	at java.io.BufferedInputStream.read(BufferedInputStream.java:344) ~[?:?]
	at sun.net.www.MeteredStream.read(MeteredStream.java:134) ~[?:?]
	at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3505) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:164) ~[?:?]
	at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:?]
	at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:49) ~[?:?]
	at com.google.api.client.util.IOUtils.copy(IOUtils.java:87) ~[?:?]
	at com.google.api.client.util.IOUtils.copy(IOUtils.java:59) ~[?:?]
	at com.google.api.client.http.HttpResponse.download(HttpResponse.java:386) ~[?:?]
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:684) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124) ~[?:?]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[?:?]
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:123) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.lambda$read$0(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
	at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.read(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:?]
	at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-7.5.1.jar:7.5.1]
	at java.io.InputStream.read(InputStream.java:213) ~[?:?]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFile(FileRestoreContext.java:240) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFiles(FileRestoreContext.java:217) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:180) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:1253) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.Repository.restoreShard(Repository.java:231) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:469) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
	Suppressed: java.io.IOException: Broken pipe
		at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:421) ~[?:?]
		at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:441) ~[?:?]
		at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:825) ~[?:?]
		at java.net.Socket$SocketOutputStream.write(Socket.java:1007) ~[?:?]
		at sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:82) ~[?:?]
		at sun.security.ssl.TransportContext.fatal(TransportContext.java:355) ~[?:?]
		at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?]
		at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:?]
		at sun.security.ssl.SSLTransport.decode(SSLTransport.java:137) ~[?:?]
		at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1463) ~[?:?]
		at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1431) ~[?:?]
		at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1022) ~[?:?]
		at java.io.BufferedInputStream.fill(BufferedInputStream.java:245) ~[?:?]
		at java.io.BufferedInputStream.read1(BufferedInputStream.java:285) ~[?:?]
		at java.io.BufferedInputStream.read(BufferedInputStream.java:344) ~[?:?]
		at sun.net.www.MeteredStream.read(MeteredStream.java:134) ~[?:?]
		at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:?]
		at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3505) ~[?:?]
		at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:164) ~[?:?]
		at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:?]
		at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:49) ~[?:?]
		at com.google.api.client.util.IOUtils.copy(IOUtils.java:87) ~[?:?]
		at com.google.api.client.util.IOUtils.copy(IOUtils.java:59) ~[?:?]
		at com.google.api.client.http.HttpResponse.download(HttpResponse.java:386) ~[?:?]
		at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:684) ~[?:?]
		at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127) ~[?:?]
		at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124) ~[?:?]
		at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[?:?]
		at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
		at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
		at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:123) ~[?:?]
		at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.lambda$read$0(GoogleCloudStorageBlobStore.java:189) ~[?:?]
		at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
		at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
		at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.read(GoogleCloudStorageBlobStore.java:189) ~[?:?]
		at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:?]
		at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:?]
		at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:?]
		at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-7.5.1.jar:7.5.1]
		at java.io.InputStream.read(InputStream.java:213) ~[?:?]
		at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFile(FileRestoreContext.java:240) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFiles(FileRestoreContext.java:217) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:180) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:1253) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.repositories.Repository.restoreShard(Repository.java:231) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:469) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.1.jar:7.5.1]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
		at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.io.IOException: Connection reset
	at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:324) ~[?:?]
	at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:351) ~[?:?]
	at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:802) ~[?:?]
	at java.net.Socket$SocketInputStream.read(Socket.java:937) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:450) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:237) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:190) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:108) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1463) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1431) ~[?:?]
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1022) ~[?:?]
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:245) ~[?:?]
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:285) ~[?:?]
	at java.io.BufferedInputStream.read(BufferedInputStream.java:344) ~[?:?]
	at sun.net.www.MeteredStream.read(MeteredStream.java:134) ~[?:?]
	at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3505) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:164) ~[?:?]
	at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:?]
	at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:49) ~[?:?]
	at com.google.api.client.util.IOUtils.copy(IOUtils.java:87) ~[?:?]
	at com.google.api.client.util.IOUtils.copy(IOUtils.java:59) ~[?:?]
	at com.google.api.client.http.HttpResponse.download(HttpResponse.java:386) ~[?:?]
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:684) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124) ~[?:?]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[?:?]
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:123) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.lambda$read$0(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
	at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.read(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:?]
	at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-7.5.1.jar:7.5.1]
	at java.io.InputStream.read(InputStream.java:213) ~[?:?]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFile(FileRestoreContext.java:240) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFiles(FileRestoreContext.java:217) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:180) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:1253) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.Repository.restoreShard(Repository.java:231) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:469) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
[2020-02-13T12:10:57,130][WARN ][o.e.i.c.IndicesClusterStateService] [elasticsearch-1] [elasticlogs-2018-05-13][0] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [elasticlogs-2018-05-13][0]: Recovery failed on {elasticsearch-1}{d2sAR621RziQbILiZMBvfg}{cgzSxj1tSjawYThZfSEXpQ}{10.10.9.72}{10.10.9.72:9300}{dim}{xpack.installed=true}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2584) [elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.1.jar:7.5.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:353) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:480) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [snapshot_80gb/fioJzVMyTcKPCRdfN5lZ7w]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:1255) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.Repository.restoreShard(Repository.java:231) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:469) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
Caused by: com.google.cloud.storage.StorageException: Connection reset
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:227) ~[?:?]
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:690) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124) ~[?:?]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[?:?]
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:123) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.lambda$read$0(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
	at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.read(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:?]
	at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-7.5.1.jar:7.5.1]
	at java.io.InputStream.read(InputStream.java:213) ~[?:?]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFile(FileRestoreContext.java:240) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFiles(FileRestoreContext.java:217) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:180) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:1253) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.Repository.restoreShard(Repository.java:231) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:469) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
Caused by: javax.net.ssl.SSLException: Connection reset
	at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:137) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1463) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1431) ~[?:?]
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1022) ~[?:?]
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:245) ~[?:?]
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:285) ~[?:?]
	at java.io.BufferedInputStream.read(BufferedInputStream.java:344) ~[?:?]
	at sun.net.www.MeteredStream.read(MeteredStream.java:134) ~[?:?]
	at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3505) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:164) ~[?:?]
	at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:?]
	at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:49) ~[?:?]
	at com.google.api.client.util.IOUtils.copy(IOUtils.java:87) ~[?:?]
	at com.google.api.client.util.IOUtils.copy(IOUtils.java:59) ~[?:?]
	at com.google.api.client.http.HttpResponse.download(HttpResponse.java:386) ~[?:?]
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:684) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124) ~[?:?]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[?:?]
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:123) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.lambda$read$0(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
	at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.read(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:?]
	at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-7.5.1.jar:7.5.1]
	at java.io.InputStream.read(InputStream.java:213) ~[?:?]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFile(FileRestoreContext.java:240) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFiles(FileRestoreContext.java:217) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:180) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:1253) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.Repository.restoreShard(Repository.java:231) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:469) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more
	Suppressed: java.net.SocketException: Broken pipe
		at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:421) ~[?:?]
		at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:441) ~[?:?]
		at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:825) ~[?:?]
		at java.net.Socket$SocketOutputStream.write(Socket.java:1007) ~[?:?]
		at sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:82) ~[?:?]
		at sun.security.ssl.TransportContext.fatal(TransportContext.java:355) ~[?:?]
		at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?]
		at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:?]
		at sun.security.ssl.SSLTransport.decode(SSLTransport.java:137) ~[?:?]
		at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1463) ~[?:?]
		at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1431) ~[?:?]
		at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1022) ~[?:?]
		at java.io.BufferedInputStream.fill(BufferedInputStream.java:245) ~[?:?]
		at java.io.BufferedInputStream.read1(BufferedInputStream.java:285) ~[?:?]
		at java.io.BufferedInputStream.read(BufferedInputStream.java:344) ~[?:?]
		at sun.net.www.MeteredStream.read(MeteredStream.java:134) ~[?:?]
		at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:?]
		at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3505) ~[?:?]
		at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:164) ~[?:?]
		at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:?]
		at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:49) ~[?:?]
		at com.google.api.client.util.IOUtils.copy(IOUtils.java:87) ~[?:?]
		at com.google.api.client.util.IOUtils.copy(IOUtils.java:59) ~[?:?]
		at com.google.api.client.http.HttpResponse.download(HttpResponse.java:386) ~[?:?]
		at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:684) ~[?:?]
		at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127) ~[?:?]
		at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124) ~[?:?]
		at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[?:?]
		at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
		at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
		at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:123) ~[?:?]
		at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.lambda$read$0(GoogleCloudStorageBlobStore.java:189) ~[?:?]
		at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
		at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
		at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.read(GoogleCloudStorageBlobStore.java:189) ~[?:?]
		at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:?]
		at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:?]
		at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:?]
		at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-7.5.1.jar:7.5.1]
		at java.io.InputStream.read(InputStream.java:213) ~[?:?]
		at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFile(FileRestoreContext.java:240) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFiles(FileRestoreContext.java:217) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:180) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:1253) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.repositories.Repository.restoreShard(Repository.java:231) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:469) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) [elasticsearch-7.5.1.jar:7.5.1]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.1.jar:7.5.1]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
		at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.net.SocketException: Connection reset
	at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:324) ~[?:?]
	at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:351) ~[?:?]
	at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:802) ~[?:?]
	at java.net.Socket$SocketInputStream.read(Socket.java:937) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:450) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:237) ~[?:?]
	at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:190) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:108) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1463) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1431) ~[?:?]
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1022) ~[?:?]
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:245) ~[?:?]
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:285) ~[?:?]
	at java.io.BufferedInputStream.read(BufferedInputStream.java:344) ~[?:?]
	at sun.net.www.MeteredStream.read(MeteredStream.java:134) ~[?:?]
	at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3505) ~[?:?]
	at com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:164) ~[?:?]
	at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:?]
	at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:49) ~[?:?]
	at com.google.api.client.util.IOUtils.copy(IOUtils.java:87) ~[?:?]
	at com.google.api.client.util.IOUtils.copy(IOUtils.java:59) ~[?:?]
	at com.google.api.client.http.HttpResponse.download(HttpResponse.java:386) ~[?:?]
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:684) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124) ~[?:?]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[?:?]
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
	at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:123) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.lambda$read$0(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
	at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
	at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore$1.read(GoogleCloudStorageBlobStore.java:189) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:?]
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:?]
	at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92) ~[elasticsearch-7.5.1.jar:7.5.1]
	at java.io.InputStream.read(InputStream.java:213) ~[?:?]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFile(FileRestoreContext.java:240) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restoreFiles(FileRestoreContext.java:217) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:180) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:1253) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.repositories.Repository.restoreShard(Repository.java:231) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:469) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$5(StoreRecovery.java:285) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:283) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1867) ~[elasticsearch-7.5.1.jar:7.5.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$17(IndexShard.java:2580) ~[elasticsearch-7.5.1.jar:7.5.1]
	... 4 more

l.e. I've seen java.io.IOException: Connection reset and javax.net.ssl.SSLException: Connection reset, neither of which seem to be retryable

Relates #46589, which works around the same issue in the AWS S3 SDK.

Quoting googleapis/java-storage#26 (comment):

network issues [...] are out of the scope of its responsibility

This seems to ignore the fact that connection drops are often transient and resuming the download is often the right way to proceed. Still, at least it's a clear statement of intent, and indicates that maybe we need to implement our own resume functionality to work around this for now.
@DaveCTurner DaveCTurner added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs labels Feb 13, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

@original-brownbear original-brownbear self-assigned this Feb 18, 2020
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Feb 18, 2020
@original-brownbear
Add Blob Download Retries to GCS Repository
e6815fa 
Exactly as elastic#46589 (and kept as close to it as possible code wise so we can dry things up in a follow-up potentially) but for GCS.

Closes elastic#52319
@original-brownbear original-brownbear mentioned this issue Feb 18, 2020
Merged
original-brownbear added a commit that referenced this issue Feb 19, 2020
@original-brownbear
Add Blob Download Retries to GCS Repository (#52479)
c665cf0 
* Add Blob Download Retries to GCS Repository

Exactly as #46589 (and kept as close to it as possible code wise so we can dry things up in a follow-up potentially) but for GCS.

Closes #52319
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Feb 19, 2020
@original-brownbear
Add Blob Download Retries to GCS Repository (elastic#52479)
4740437 
* Add Blob Download Retries to GCS Repository

Exactly as elastic#46589 (and kept as close to it as possible code wise so we can dry things up in a follow-up potentially) but for GCS.

Closes elastic#52319
@original-brownbear original-brownbear mentioned this issue Feb 19, 2020
Merged
sbourke pushed a commit to sbourke/elasticsearch that referenced this issue Feb 19, 2020
@original-brownbear @sbourke
Add Blob Download Retries to GCS Repository (elastic#52479)
70d4484 
* Add Blob Download Retries to GCS Repository

Exactly as elastic#46589 (and kept as close to it as possible code wise so we can dry things up in a follow-up potentially) but for GCS.

Closes elastic#52319
original-brownbear added a commit that referenced this issue Feb 19, 2020
@original-brownbear
Add Blob Download Retries to GCS Repository (#52479) (#52521)
aeb7b77 
* Add Blob Download Retries to GCS Repository

Exactly as #46589 (and kept as close to it as possible code wise so we can dry things up in a follow-up potentially) but for GCS.

Closes #52319
@codebrain codebrain mentioned this issue Apr 1, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add Blob Download Retries to GCS Repository original-brownbear/elasticsearch
3 participants
@DaveCTurner @original-brownbear @elasticmachine