Introduce repository test kit/analyser

docs/reference/snapshot-restore/apis/repo-speed-test-api.asciidoc

@@ -0,0 +1,127 @@
+[role="xpack"]
+[[repo-speed-test-api]]
+=== Repository speed test API
+++++
+<titleabbrev>Repository speed test</titleabbrev>
+++++
+
+Measures the performance characteristics of a snapshot repository.
+
+////
+[source,console]
+----
+PUT /_snapshot/my_repository
+{
+  "type": "fs",
+  "settings": {
+    "location": "my_backup_location"
+  }
+}
+----
+// TESTSETUP
+////
+
+[source,console]
+----
+POST /_snapshot/my_repository/_speed_test?blob_count=10&concurrency=4&max_blob_size=1mb&timeout=120s
+----
+
+[[repo-speed-test-api-request]]
+==== {api-request-title}
+
+`POST /_snapshot/<repository>/_speed_test`
+
+[[repo-speed-test-api-desc]]
+==== {api-description-title}
+
+There are a large number of third-party storage systems available, not all of
+which are suitable for use as a snapshot repository by {es}. Some storage
+systems perform poorly, or behave incorrectly, especially when accessed
+concurrently by multiple clients as the nodes of an {es} cluster do.
+
+The Repository speed test API performs a collection of read and write
+operations on your repository which are specially designed to detect incorrect
+behaviour and to measure the performance characteristics of your storage
+system.
+
+Each speed test runs a wide variety of operations generated by a pseudo-random
+process. You can seed this process using the optional `seed` parameter in order
+to repeat the same set of operations in multiple experiments. Note that the
+operations are performed concurrently so may not always happen in the same
+order on each run.
+
+The default values for the parameters to this API are deliberately low to
+reduce the impact of running this API accidentally. A realistic experiment
+should set `blob_count` to at least `2000` and `max_blob_size` to at least
+`2gb`, and will almost certainly need to increase the `timeout` to allow time
+for the process to complete successfully.
+
+If the speed test is successful this API returns details of the testing
+process, including how long each operation took. You can use this information
+to analyse the performance of your storage system. If any operation fails or
+returns an incorrect result, this API returns an error. If the API returns an
+error then it may not have removed all the data it wrote to the repository. The
+error will indicate the location of any leftover data, and this path is also
+recorded in the {es} logs. You should verify yourself that this location has
+been cleaned up correctly. If there is still leftover data at the specified
+location then you should manually remove it.
+
+If the connection from your client to {es} is closed while the client is
+waiting for the result of the speed test then the test is cancelled.  Since a
+speed test takes a long time to complete, you may need to configure your client
+to wait for longer than usual for a response. On cancellation the speed test
+attempts to clean up the data it was writing, but it may not be able to remove
+it all. The path to the leftover data is recorded in the {es} logs. You should
+verify yourself that this location has been cleaned up correctly. If there is
+still leftover data at the specified location then you should manually remove
+it.
+
+NOTE: A speed test writes a substantial amount of data to your repository and
+then reads it back again. This consumes bandwidth on the network between the
+cluster and the repository, and storage space and IO bandwidth on the
+repository itself. You must ensure this load does not affect other users of
+these systems. Speed tests respect the repository settings
+`max_snapshot_bytes_per_sec` and `max_restore_bytes_per_sec` if available, and
+the cluster setting `indices.recovery.max_bytes_per_sec` which you can use to
+limit the bandwidth they consume.
+
+[[repo-speed-test-api-path-params]]
+==== {api-path-parms-title}
+
+`<repository>`::
+(Required, string)
+Name of the snapshot repository to test.
+
+[[repo-speed-test-api-query-params]]
+==== {api-query-parms-title}
+
+`blob_count`::
+(Optional, integer) The total number of blobs to write to the repository during
+the test. Defaults to `100`. For realistic experiments you should set this to
+at least `2000`.
+
+`concurrency`::
+(Optional, integer) The number of write operations to perform concurrently.
+Defaults to `10`.
+
+`seed`::
+(Optional, integer) The seed for the pseudo-random number generator used to
+generate the list of operations performed during the test. To repeat the same
+set of operations in multiple experiments, use the same seed in each
+experiment.
+
+`max_blob_size`::
+(Optional, <<size-units, size units>>) The maximum size of a blob to be written
+during the test. Defaults to `10mb`. For realistic experiments you should set
+this to at least `2gb`.
+
+`timeout`::
+(Optional, <<time-units, time units>>) Specifies the period of time to wait for
+the test to complete. If no response is received before the timeout expires,
+the test is cancelled and returns an error. Defaults to `30s`.
+
+[role="child_attributes"]
+[[repo-speed-test-api-response-body]]
+==== {api-response-body-title}
+
+TODO

docs/reference/snapshot-restore/apis/snapshot-restore-apis.asciidoc

@@ -27,6 +27,7 @@ For more information, see <<snapshot-restore>>.
 
 include::put-repo-api.asciidoc[]
 include::verify-repo-api.asciidoc[]
+include::repo-speed-test-api.asciidoc[]
 include::get-repo-api.asciidoc[]
 include::delete-repo-api.asciidoc[]
 include::clean-up-repo-api.asciidoc[]

docs/reference/snapshot-restore/register-repository.asciidoc

@@ -258,6 +258,9 @@ POST /_snapshot/my_unverified_backup/_verify
 
 It returns a list of nodes where repository was successfully verified or an error message if verification process failed.
 
+If desired, you can also test a repository more thoroughly using the
+<<repo-speed-test-api>>.
+
 [discrete]
 [[snapshots-repository-cleanup]]
 === Repository cleanup

server/src/main/java/org/elasticsearch/repositories/RepositoriesService.java

@@ -284,7 +284,7 @@ protected void doRun() {
         });
     }
 
-    static boolean isDedicatedVotingOnlyNode(Set<DiscoveryNodeRole> roles) {
+    public static boolean isDedicatedVotingOnlyNode(Set<DiscoveryNodeRole> roles) {
         return roles.contains(DiscoveryNodeRole.MASTER_ROLE) && roles.contains(DiscoveryNodeRole.DATA_ROLE) == false &&
             roles.stream().anyMatch(role -> role.roleName().equals("voting_only"));
     }

server/src/main/java/org/elasticsearch/repositories/RepositoryVerificationException.java

@@ -46,5 +46,11 @@ public RestStatus status() {
     public RepositoryVerificationException(StreamInput in) throws IOException{
         super(in);
     }
+
+    @Override
+    public synchronized Throwable fillInStackTrace() {
+        // stack trace for a verification failure is uninteresting, the message has all the information we need
+        return this;
+    }
 }
 

server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java

@@ -2289,17 +2289,47 @@ private static ActionListener<Void> fileQueueListener(BlockingQueue<BlobStoreInd
         });
     }
 
-    private static InputStream maybeRateLimit(InputStream stream, Supplier<RateLimiter> rateLimiterSupplier, CounterMetric metric) {
-        return new RateLimitingInputStream(stream, rateLimiterSupplier, metric::inc);
+    private static InputStream maybeRateLimit(
+            InputStream stream,
+            Supplier<RateLimiter> rateLimiterSupplier,
+            RateLimitingInputStream.Listener throttleListener) {
+        return new RateLimitingInputStream(stream, rateLimiterSupplier, throttleListener);
     }
 
+    /**
+     * Wrap the restore rate limiter (controlled by the repository setting `max_restore_bytes_per_sec` and the cluster setting
+     * `indices.recovery.max_bytes_per_sec`) around the given stream. Any throttling is reported to the given listener and not otherwise
+     * recorded in the value returned by {@link BlobStoreRepository#getRestoreThrottleTimeInNanos}.
+     */
     public InputStream maybeRateLimitRestores(InputStream stream) {
-        return maybeRateLimit(maybeRateLimit(stream, () -> restoreRateLimiter, restoreRateLimitingTimeInNanos),
-            recoverySettings::rateLimiter, restoreRateLimitingTimeInNanos);
+        return maybeRateLimitRestores(stream, restoreRateLimitingTimeInNanos::inc);
     }
 
+    /**
+     * Wrap the restore rate limiter (controlled by the repository setting `max_restore_bytes_per_sec` and the cluster setting
+     * `indices.recovery.max_bytes_per_sec`) around the given stream. Any throttling is recorded in the value returned by {@link
+     * BlobStoreRepository#getRestoreThrottleTimeInNanos}.
+     */
+    public InputStream maybeRateLimitRestores(InputStream stream, RateLimitingInputStream.Listener throttleListener) {
+        return maybeRateLimit(maybeRateLimit(stream, () -> restoreRateLimiter, throttleListener),
+            recoverySettings::rateLimiter, throttleListener);
+    }
+
+    /**
+     * Wrap the snapshot rate limiter (controlled by the repository setting `max_snapshot_bytes_per_sec`) around the given stream. Any
+     * throttling is recorded in the value returned by {@link BlobStoreRepository#getSnapshotThrottleTimeInNanos()}.
+     */
     public InputStream maybeRateLimitSnapshots(InputStream stream) {
-        return maybeRateLimit(stream, () -> snapshotRateLimiter, snapshotRateLimitingTimeInNanos);
+        return maybeRateLimitSnapshots(stream, snapshotRateLimitingTimeInNanos::inc);
+    }
+
+    /**
+     * Wrap the snapshot rate limiter (controlled by the repository setting `max_snapshot_bytes_per_sec`) around the given stream. Any
+     * throttling is reported to the given listener and not otherwise recorded in the value returned by {@link
+     * BlobStoreRepository#getSnapshotThrottleTimeInNanos()}.
+     */
+    public InputStream maybeRateLimitSnapshots(InputStream stream, RateLimitingInputStream.Listener throttleListener) {
+        return maybeRateLimit(stream, () -> snapshotRateLimiter, throttleListener);
     }
 
     @Override
@@ -2539,6 +2569,10 @@ private static void failStoreIfCorrupted(Store store, Exception e) {
         }
     }
 
+    public boolean supportURLRepo() {
+        return supportURLRepo;
+    }
+
     /**
      * The result of removing a snapshot from a shard folder in the repository.
      */

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/operator/OperatorOnlyRegistry.java

@@ -24,7 +24,12 @@ public class OperatorOnlyRegistry {
         "cluster:admin/autoscaling/put_autoscaling_policy",
         "cluster:admin/autoscaling/delete_autoscaling_policy",
         "cluster:admin/autoscaling/get_autoscaling_policy",
-        "cluster:admin/autoscaling/get_autoscaling_capacity");
+        "cluster:admin/autoscaling/get_autoscaling_capacity",
+        // Repository speed test actions are not mentioned in core, literal strings are needed.
+        "cluster:admin/repository/speed_test",
+        "cluster:admin/repository/speed_test/blob",
+        "cluster:admin/repository/speed_test/blob/read"
+        );
 
     /**
      * Check whether the given action and request qualify as operator-only. The method returns

x-pack/plugin/snapshot-repo-test-kit/build.gradle

@@ -0,0 +1,30 @@
+apply plugin: 'elasticsearch.internal-cluster-test'
+apply plugin: 'elasticsearch.esplugin'
+esplugin {
+  name 'snapshot-repo-test-kit'
+  description 'A plugin for a test kit for snapshot repositories'
+  classname 'org.elasticsearch.repositories.blobstore.testkit.SnapshotRepositoryTestKit'
+  extendedPlugins = ['x-pack-core']
+}
+archivesBaseName = 'x-pack-snapshot-repo-test-kit'
+
+dependencies {
+  compileOnly project(path: xpackModule('core'), configuration: 'default')
+  internalClusterTestImplementation project(path: xpackModule('core'), configuration: 'testArtifacts')
+}
+
+addQaCheckDependencies()
+
+configurations {
+  testArtifacts.extendsFrom testRuntime
+  testArtifacts.extendsFrom testImplementation
+}
+
+def testJar = tasks.register("testJar", Jar) {
+  appendix 'test'
+  from sourceSets.test.output
+}
+
+artifacts {
+  testArtifacts testJar
+}

x-pack/plugin/snapshot-repo-test-kit/qa/build.gradle

@@ -0,0 +1,6 @@
+apply plugin: 'elasticsearch.build'
+tasks.named("test").configure { enabled = false }
+
+dependencies {
+  api project(':test:framework')
+}

x-pack/plugin/snapshot-repo-test-kit/qa/rest/build.gradle

@@ -0,0 +1,26 @@
+apply plugin: 'elasticsearch.testclusters'
+apply plugin: 'elasticsearch.standalone-rest-test'
+apply plugin: 'elasticsearch.rest-test'
+apply plugin: 'elasticsearch.rest-resources'
+
+dependencies {
+  testImplementation project(path: xpackModule('snapshot-repo-test-kit'), configuration: 'testArtifacts')
+}
+
+final File repoDir = file("$buildDir/testclusters/repo")
+
+tasks.named("integTest").configure {
+  systemProperty 'tests.path.repo', repoDir
+}
+
+testClusters.matching { it.name == "integTest" }.configureEach {
+  testDistribution = 'DEFAULT'
+  setting 'path.repo', repoDir.absolutePath
+}
+
+restResources {
+  restApi {
+    includeCore 'indices', 'search', 'bulk', 'snapshot', 'nodes', '_common'
+    includeXpack 'snapshot_repo_test_kit'
+  }
+}

x-pack/plugin/snapshot-repo-test-kit/qa/rest/src/test/java/org/elasticsearch/repositories/blobstore/testkit/rest/FsSnapshotRepoTestKitIT.java

@@ -0,0 +1,24 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit.rest;
+
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.repositories.blobstore.testkit.AbstractSnapshotRepoTestKitRestTestCase;
+import org.elasticsearch.repositories.fs.FsRepository;
+
+public class FsSnapshotRepoTestKitIT extends AbstractSnapshotRepoTestKitRestTestCase {
+
+    @Override
+    protected String repositoryType() {
+        return FsRepository.TYPE;
+    }
+
+    @Override
+    protected Settings repositorySettings() {
+        return Settings.builder().put("location", System.getProperty("tests.path.repo")).build();
+    }
+}

x-pack/plugin/snapshot-repo-test-kit/qa/rest/src/test/java/org/elasticsearch/repositories/blobstore/testkit/rest/SnapshotRepoTestKitClientYamlTestSuiteIT.java

@@ -0,0 +1,23 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+package org.elasticsearch.repositories.blobstore.testkit.rest;
+
+import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
+import org.elasticsearch.test.rest.yaml.ClientYamlTestCandidate;
+import org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase;
+
+public class SnapshotRepoTestKitClientYamlTestSuiteIT extends ESClientYamlSuiteTestCase {
+
+    public SnapshotRepoTestKitClientYamlTestSuiteIT(final ClientYamlTestCandidate testCandidate) {
+        super(testCandidate);
+    }
+
+    @ParametersFactory
+    public static Iterable<Object[]> parameters() throws Exception {
+        return ESClientYamlSuiteTestCase.createParameters();
+    }
+}