Make FsBlobContainer Listing Resilient to Concurrent Modifications (#49142) (#49176)

* Make FsBlobContainer Listing Resilient to Concurrent Modifications

If we list out files in a folder via the lazily computed directory
stream, we have to deal with concurrent deletes when reading the file
attributes since we don't have a lock on the directory in any way.

Closes #37581

Author: original-brownbear

server/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java

@@ -97,7 +97,13 @@ public Map<String, BlobMetaData> listBlobsByPrefix(String blobNamePrefix) throws
         blobNamePrefix = blobNamePrefix == null ? "" : blobNamePrefix;
         try (DirectoryStream<Path> stream = Files.newDirectoryStream(path, blobNamePrefix + "*")) {
             for (Path file : stream) {
-                final BasicFileAttributes attrs = Files.readAttributes(file, BasicFileAttributes.class);
+                final BasicFileAttributes attrs;
+                try {
+                    attrs = Files.readAttributes(file, BasicFileAttributes.class);
+                } catch (FileNotFoundException | NoSuchFileException e) {
+                    // The file was concurrently deleted between listing files and trying to get its attributes so we skip it here
+                    continue;
+                }
                 if (attrs.isRegularFile()) {
                     builder.put(file.getFileName().toString(), new PlainBlobMetaData(file.getFileName().toString(), attrs.size()));
                 }