Ensures pending index-* blobs are deleted when snapshotting. The

index-* blobs are generational files that maintain the snapshots
in the repository.  To write these atomically, we first write a
`pending-index-*` blob, then move it to `index-*`, which also deletes
`pending-index-*` in case its not a file-system level move (e.g.
S3 repositories) .  For example, to write the 5th generation of the
index blob for the repository, we would first write the bytes to
`pending-index-5` and then move `pending-index-5` to `index-5`.  It is
possible that we fail after writing `pending-index-5`, but before
moving it to `index-5` or deleting `pending-index-5`.  In this case,
we will have a dangling `pending-index-5` blob laying around.  Since
snapshot elastic#5 would have failed, the next snapshot assumes a generation
number of 5, so it tries to write to `index-5`, which first tries to
write to `pending-index-5` before moving the blob to `index-5`.  Since
`pending-index-5` is leftover from the previous failure, the snapshot
fails as it cannot overwrite this blob.

This commit solves the problem by first, adding a UUID to the
`pending-index-*` blobs, and secondly, strengthen the logic around
failure to write the `index-*` generational blob to ensure pending
files are deleted on cleanup.

Closes elastic#21462

Author: Ali Beyad

core/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java

@@ -867,16 +867,23 @@ private long listBlobsToGetLatestIndexId() throws IOException {
     }
 
     private void writeAtomic(final String blobName, final BytesReference bytesRef) throws IOException {
-        final String tempBlobName = "pending-" + blobName;
+        final String tempBlobName = "pending-" + UUIDs.randomBase64UUID() + "-" + blobName;
         try (InputStream stream = bytesRef.streamInput()) {
             snapshotsBlobContainer.writeBlob(tempBlobName, stream, bytesRef.length());
-        }
-        try {
             snapshotsBlobContainer.move(tempBlobName, blobName);
         } catch (IOException ex) {
-            // Move failed - try cleaning up
-            snapshotsBlobContainer.deleteBlob(tempBlobName);
-            throw ex;
+            IOException exceptionToThrow = ex;
+            try {
+                snapshotsBlobContainer.deleteBlob(tempBlobName);
+            } catch (NoSuchFileException e) {
+                // the temp file does not exist, so writeBlob probably failed above,
+                // exceptionToThrow will capture this and be thrown at the end
+            } catch (IOException e) {
+                if (exceptionToThrow == null) {
+                    exceptionToThrow = e;
+                }
+            }
+            throw exceptionToThrow;
         }
     }