Add retention to Snapshot Lifecycle Management (#46407)

This commit adds retention to the existing Snapshot Lifecycle Management feature (#38461) as described in #43663. This allows a user to configure SLM to automatically delete older snapshots based on a number of criteria.

An example policy would look like:

```
PUT /_slm/policy/snapshot-every-day
{
  "schedule": "0 30 2 * * ?",
  "name": "<production-snap-{now/d}>",
  "repository": "my-s3-repository",
  "config": {
    "indices": ["foo-*", "important"]
  },
  // Newly configured retention options
  "retention": {
    // Snapshots should be deleted after 14 days
    "expire_after": "14d",
    // Keep a maximum of thirty snapshots
    "max_count": 30,
    // Keep a minimum of the four most recent snapshots
    "min_count": 4
  }
}
```

SLM Retention is run on a scheduled configurable with the `slm.retention_schedule` setting, which supports cron expressions. Deletions are run for a configurable time bounded by the `slm.retention_duration` setting, which defaults to 1 hour.

Included in this work is a new SLM stats API endpoint available through

``` json
GET /_slm/stats
```

That returns statistics about snapshot taken and deleted, as well as successful retention runs, failures, and the time spent deleting snapshots. #45362 has more information as well as an example of the output. These stats are also included when retrieving SLM policies via the API.

* Add base framework for snapshot retention (#43605)

* Add base framework for snapshot retention

This adds a basic `SnapshotRetentionService` and `SnapshotRetentionTask`
to start as the basis for SLM's retention implementation.

Relates to #38461

* Remove extraneous 'public'

* Use a local var instead of reading class var repeatedly

* Add SnapshotRetentionConfiguration for retention configuration (#43777)

* Add SnapshotRetentionConfiguration for retention configuration

This commit adds the `SnapshotRetentionConfiguration` class and its HLRC
counterpart to encapsulate the configuration for SLM retention.
Currently only a single parameter is supported as an example (we still
need to discuss the different options we want to support and their
names) to keep the size of the PR down. It also does not yet include version serialization checks
since the original SLM branch has not yet been merged.

Relates to #43663

* Fix REST tests

* Fix more documentation

* Use Objects.equals to avoid NPE

* Put `randomSnapshotLifecyclePolicy` in only one place

* Occasionally return retention with no configuration

* Implement SnapshotRetentionTask's snapshot filtering and delet… (#44764)

* Implement SnapshotRetentionTask's snapshot filtering and deletion

This commit implements the snapshot filtering and deletion for
`SnapshotRetentionTask`. Currently only the expire-after age is used for
determining whether a snapshot is eligible for deletion.

Relates to #43663

* Fix deletes running on the wrong thread

* Handle missing or null policy in snap metadata differently

* Convert Tuple<String, List<SnapshotInfo>> to Map<String, List<SnapshotInfo>>

* Use the `OriginSettingClient` to work with security, enhance logging

* Prevent NPE in test by mocking Client

* Allow empty/missing SLM retention configuration (#45018)

Semi-related to #44465, this allows the `"retention"` configuration map
to be missing.

Relates to #43663

* Add min_count and max_count as SLM retention predicates (#44926)

This adds the configuration options for `min_count` and `max_count` as
well as the logic for determining whether a snapshot meets this criteria
to SLM's retention feature.

These options are optional and one, two, or all three can be specified
in an SLM policy.

Relates to #43663

* Time-bound deletion of snapshots in retention delete function (#45065)

* Time-bound deletion of snapshots in retention delete function

With a cluster that has a large number of snapshots, it's possible that
snapshot deletion can take a very long time (especially since deletes
currently have to happen in a serial fashion). To prevent snapshot
deletion from taking forever in a cluster and blocking other operations,
this commit adds a setting to allow configuring a maximum time to spend
deletion snapshots during retention. This dynamic setting defaults to 1
hour and is best-effort, meaning that it doesn't hard stop a deletion
at an hour mark, but ensures that once the time has passed, all
subsequent deletions are deferred until the next retention cycle.

Relates to #43663

* Wow snapshots suuuure can take a long time.

* Use a LongSupplier instead of actually sleeping

* Remove TestLogging annotation

* Remove rate limiting

* Add SLM metrics gathering and endpoint (#45362)

* Add SLM metrics gathering and endpoint

This commit adds the infrastructure to gather metrics about the different SLM actions that a cluster
takes. These actions are stored in `SnapshotLifecycleStats` and perpetuated in cluster state. The
stats stored include the number of snapshots taken, failed, deleted, the number of retention runs,
as well as per-policy counts for snapshots taken, failed, and deleted. It also includes the amount
of time spent deleting snapshots from SLM retention.

This commit also adds an endpoint for retrieving all stats (further commits will expose this in the
SLM get-policy API) that looks like:

```
GET /_slm/stats
{
  "retention_runs" : 13,
  "retention_failed" : 0,
  "retention_timed_out" : 0,
  "retention_deletion_time" : "1.4s",
  "retention_deletion_time_millis" : 1404,
  "policy_metrics" : {
    "daily-snapshots2" : {
      "snapshots_taken" : 7,
      "snapshots_failed" : 0,
      "snapshots_deleted" : 6,
      "snapshot_deletion_failures" : 0
    },
    "daily-snapshots" : {
      "snapshots_taken" : 12,
      "snapshots_failed" : 0,
      "snapshots_deleted" : 12,
      "snapshot_deletion_failures" : 6
    }
  },
  "total_snapshots_taken" : 19,
  "total_snapshots_failed" : 0,
  "total_snapshots_deleted" : 18,
  "total_snapshot_deletion_failures" : 6
}
```

This does not yet include HLRC for this, as this commit is quite large on its own. That will be
added in a subsequent commit.

Relates to #43663

* Version qualify serialization

* Initialize counters outside constructor

* Use computeIfAbsent instead of being too verbose

* Move part of XContent generation into subclass

* Fix REST action for master merge

* Unused import

*  Record history of SLM retention actions (#45513)

This commit records the deletion of snapshots by the retention component
of SLM into the SLM history index for the purposes of reviewing operations
taken by SLM and alerting.

* Retry SLM retention after currently running snapshot completes (#45802)

* Retry SLM retention after currently running snapshot completes

This commit adds a ClusterStateObserver to wait until the currently
running snapshot is complete before proceeding with snapshot deletion.
SLM retention waits for the maximum allowed deletion time for the
snapshot to complete, however, the waiting time is not factored into
the limit on actual deletions.

Relates to #43663

* Increase timeout waiting for snapshot completion

* Apply patch

From https://github.com/original-brownbear/elasticsearch/commit/2374316f0d1912c9e1498bece195546a1dc60bce.patch

* Rename test variables

* [TEST] Be less strict for stats checking

* Skip SLM retention if ILM is STOPPING or STOPPED (#45869)

This adds a check to ensure we take no action during SLM retention if
ILM is currently stopped or in the process of stopping.

Relates to #43663

* Check all actions preventing snapshot delete during retention (#45992)

* Check all actions preventing snapshot delete during retention run

Previously we only checked to see if a snapshot was currently running,
but it turns out that more things can block snapshot deletion. This
changes the check to be a check for:

- a snapshot currently running
- a deletion already in progress
- a repo cleanup in progress
- a restore currently running

This was found by CI where a third party delete in a test caused SLM
retention deletion to throw an exception.

Relates to #43663

* Add unit test for okayToDeleteSnapshots

* Fix bug where SLM retention task would be scheduled on every node

* Enhance test logging

* Ignore if snapshot is already deleted

* Missing import

* Fix SnapshotRetentionServiceTests

* Expose SLM policy stats in get SLM policy API (#45989)

This also adds support for the SLM stats endpoint to the high level rest client.

Retrieving a policy now looks like:

```json
{
  "daily-snapshots" : {
    "version": 1,
    "modified_date": "2019-04-23T01:30:00.000Z",
    "modified_date_millis": 1556048137314,
    "policy" : {
      "schedule": "0 30 1 * * ?",
      "name": "<daily-snap-{now/d}>",
      "repository": "my_repository",
      "config": {
        "indices": ["data-*", "important"],
        "ignore_unavailable": false,
        "include_global_state": false
      },
      "retention": {}
    },
    "stats": {
      "snapshots_taken": 0,
      "snapshots_failed": 0,
      "snapshots_deleted": 0,
      "snapshot_deletion_failures": 0
    },
    "next_execution": "2019-04-24T01:30:00.000Z",
    "next_execution_millis": 1556048160000
  }
}
```

Relates to #43663

* Rewrite SnapshotLifecycleIT as as ESIntegTestCase (#46356)

* Rewrite SnapshotLifecycleIT as as ESIntegTestCase

This commit splits `SnapshotLifecycleIT` into two different tests.
`SnapshotLifecycleRestIT` which includes the tests that do not require
slow repositories, and `SLMSnapshotBlockingIntegTests` which is now an
integration test using `MockRepository` to simulate a snapshot being in
progress.

Relates to #43663
Resolves #46205

* Add error logging when exceptions are thrown

Author: dakrone

client/rest-high-level/src/main/java/org/elasticsearch/client/IndexLifecycleClient.java

@@ -39,6 +39,8 @@
 import org.elasticsearch.client.slm.ExecuteSnapshotLifecyclePolicyResponse;
 import org.elasticsearch.client.slm.GetSnapshotLifecyclePolicyRequest;
 import org.elasticsearch.client.slm.GetSnapshotLifecyclePolicyResponse;
+import org.elasticsearch.client.slm.GetSnapshotLifecycleStatsRequest;
+import org.elasticsearch.client.slm.GetSnapshotLifecycleStatsResponse;
 import org.elasticsearch.client.slm.PutSnapshotLifecyclePolicyRequest;
 
 import java.io.IOException;
@@ -464,4 +466,40 @@ public Cancellable executeSnapshotLifecyclePolicyAsync(
             request, IndexLifecycleRequestConverters::executeSnapshotLifecyclePolicy,
             options, ExecuteSnapshotLifecyclePolicyResponse::fromXContent, listener, emptySet());
     }
+
+    /**
+     * Retrieve snapshot lifecycle statistics.
+     * See <pre>
+     *  https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/
+     *  java-rest-high-ilm-slm-get-snapshot-lifecycle-stats.html
+     * </pre>
+     * for more.
+     * @param request the request
+     * @param options the request options (e.g. headers), use {@link RequestOptions#DEFAULT} if nothing needs to be customized
+     * @return the response
+     * @throws IOException in case there is a problem sending the request or parsing back the response
+     */
+    public GetSnapshotLifecycleStatsResponse getSnapshotLifecycleStats(GetSnapshotLifecycleStatsRequest request,
+                                                                       RequestOptions options) throws IOException {
+        return restHighLevelClient.performRequestAndParseEntity(request, IndexLifecycleRequestConverters::getSnapshotLifecycleStats,
+            options, GetSnapshotLifecycleStatsResponse::fromXContent, emptySet());
+    }
+
+    /**
+     * Asynchronously retrieve snapshot lifecycle statistics.
+     * See <pre>
+     *  https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/
+     *  java-rest-high-ilm-slm-get-snapshot-lifecycle-stats.html
+     * </pre>
+     * for more.
+     * @param request the request
+     * @param options the request options (e.g. headers), use {@link RequestOptions#DEFAULT} if nothing needs to be customized
+     * @param listener the listener to be notified upon request completion
+     * @return cancellable that may be used to cancel the request
+     */
+    public Cancellable getSnapshotLifecycleStatsAsync(GetSnapshotLifecycleStatsRequest request, RequestOptions options,
+                                                      ActionListener<GetSnapshotLifecycleStatsResponse> listener) {
+        return restHighLevelClient.performRequestAsyncAndParseEntity(request, IndexLifecycleRequestConverters::getSnapshotLifecycleStats,
+            options, GetSnapshotLifecycleStatsResponse::fromXContent, listener, emptySet());
+    }
 }

client/rest-high-level/src/main/java/org/elasticsearch/client/IndexLifecycleRequestConverters.java

@@ -35,6 +35,7 @@
 import org.elasticsearch.client.slm.DeleteSnapshotLifecyclePolicyRequest;
 import org.elasticsearch.client.slm.ExecuteSnapshotLifecyclePolicyRequest;
 import org.elasticsearch.client.slm.GetSnapshotLifecyclePolicyRequest;
+import org.elasticsearch.client.slm.GetSnapshotLifecycleStatsRequest;
 import org.elasticsearch.client.slm.PutSnapshotLifecyclePolicyRequest;
 import org.elasticsearch.common.Strings;
 
@@ -215,4 +216,14 @@ static Request executeSnapshotLifecyclePolicy(ExecuteSnapshotLifecyclePolicyRequ
         request.addParameters(params.asMap());
         return request;
     }
+
+    static Request getSnapshotLifecycleStats(GetSnapshotLifecycleStatsRequest getSnapshotLifecycleStatsRequest) {
+        String endpoint = new RequestConverters.EndpointBuilder().addPathPartAsIs("_slm/stats").build();
+        Request request = new Request(HttpGet.METHOD_NAME, endpoint);
+        RequestConverters.Params params = new RequestConverters.Params();
+        params.withMasterTimeout(getSnapshotLifecycleStatsRequest.masterNodeTimeout());
+        params.withTimeout(getSnapshotLifecycleStatsRequest.timeout());
+        request.addParameters(params.asMap());
+        return request;
+    }
 }

client/rest-high-level/src/main/java/org/elasticsearch/client/slm/GetSnapshotLifecycleStatsRequest.java

@@ -0,0 +1,30 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.client.slm;
+
+import org.elasticsearch.client.TimedRequest;
+
+public class GetSnapshotLifecycleStatsRequest extends TimedRequest {
+
+    public GetSnapshotLifecycleStatsRequest() {
+        super();
+    }
+
+}

client/rest-high-level/src/main/java/org/elasticsearch/client/slm/GetSnapshotLifecycleStatsResponse.java

@@ -0,0 +1,68 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.elasticsearch.client.slm;
+
+import org.elasticsearch.common.xcontent.ToXContentObject;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.common.xcontent.XContentParser;
+
+import java.io.IOException;
+import java.util.Objects;
+
+public class GetSnapshotLifecycleStatsResponse implements ToXContentObject {
+
+    private final SnapshotLifecycleStats stats;
+
+    public GetSnapshotLifecycleStatsResponse(SnapshotLifecycleStats stats) {
+        this.stats = stats;
+    }
+
+    public SnapshotLifecycleStats getStats() {
+        return this.stats;
+    }
+
+    public static GetSnapshotLifecycleStatsResponse fromXContent(XContentParser parser) throws IOException {
+        return new GetSnapshotLifecycleStatsResponse(SnapshotLifecycleStats.parse(parser));
+    }
+
+    @Override
+    public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
+        return stats.toXContent(builder, params);
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+
+        if (o == null || getClass() != o.getClass()) {
+            return false;
+        }
+
+        GetSnapshotLifecycleStatsResponse other = (GetSnapshotLifecycleStatsResponse) o;
+        return Objects.equals(this.stats, other.stats);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(this.stats);
+    }
+}

client/rest-high-level/src/main/java/org/elasticsearch/client/slm/SnapshotLifecyclePolicy.java

@@ -38,11 +38,13 @@ public class SnapshotLifecyclePolicy implements ToXContentObject {
     private final String schedule;
     private final String repository;
     private final Map<String, Object> configuration;
+    private final SnapshotRetentionConfiguration retentionPolicy;
 
     private static final ParseField NAME = new ParseField("name");
     private static final ParseField SCHEDULE = new ParseField("schedule");
     private static final ParseField REPOSITORY = new ParseField("repository");
     private static final ParseField CONFIG = new ParseField("config");
+    private static final ParseField RETENTION = new ParseField("retention");
 
     @SuppressWarnings("unchecked")
     private static final ConstructingObjectParser<SnapshotLifecyclePolicy, String> PARSER =
@@ -52,23 +54,27 @@ public class SnapshotLifecyclePolicy implements ToXContentObject {
                 String schedule = (String) a[1];
                 String repo = (String) a[2];
                 Map<String, Object> config = (Map<String, Object>) a[3];
-                return new SnapshotLifecyclePolicy(id, name, schedule, repo, config);
+                SnapshotRetentionConfiguration retention = (SnapshotRetentionConfiguration) a[4];
+                return new SnapshotLifecyclePolicy(id, name, schedule, repo, config, retention);
             });
 
     static {
         PARSER.declareString(ConstructingObjectParser.constructorArg(), NAME);
         PARSER.declareString(ConstructingObjectParser.constructorArg(), SCHEDULE);
         PARSER.declareString(ConstructingObjectParser.constructorArg(), REPOSITORY);
         PARSER.declareObject(ConstructingObjectParser.optionalConstructorArg(), (p, c) -> p.map(), CONFIG);
+        PARSER.declareObject(ConstructingObjectParser.optionalConstructorArg(), SnapshotRetentionConfiguration::parse, RETENTION);
     }
 
     public SnapshotLifecyclePolicy(final String id, final String name, final String schedule,
-                                   final String repository, @Nullable Map<String, Object> configuration) {
-        this.id = Objects.requireNonNull(id);
-        this.name = name;
-        this.schedule = schedule;
-        this.repository = repository;
+                                   final String repository, @Nullable final Map<String, Object> configuration,
+                                   @Nullable final SnapshotRetentionConfiguration retentionPolicy) {
+        this.id = Objects.requireNonNull(id, "policy id is required");
+        this.name = Objects.requireNonNull(name, "policy snapshot name is required");
+        this.schedule = Objects.requireNonNull(schedule, "policy schedule is required");
+        this.repository = Objects.requireNonNull(repository, "policy snapshot repository is required");
         this.configuration = configuration;
+        this.retentionPolicy = retentionPolicy;
     }
 
     public String getId() {
@@ -92,6 +98,11 @@ public Map<String, Object> getConfig() {
         return this.configuration;
     }
 
+    @Nullable
+    public SnapshotRetentionConfiguration getRetentionPolicy() {
+        return this.retentionPolicy;
+    }
+
     public static SnapshotLifecyclePolicy parse(XContentParser parser, String id) {
         return PARSER.apply(parser, id);
     }
@@ -105,13 +116,16 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         if (this.configuration != null) {
             builder.field(CONFIG.getPreferredName(), this.configuration);
         }
+        if (this.retentionPolicy != null) {
+            builder.field(RETENTION.getPreferredName(), this.retentionPolicy);
+        }
         builder.endObject();
         return builder;
     }
 
     @Override
     public int hashCode() {
-        return Objects.hash(id, name, schedule, repository, configuration);
+        return Objects.hash(id, name, schedule, repository, configuration, retentionPolicy);
     }
 
     @Override
@@ -128,7 +142,8 @@ public boolean equals(Object obj) {
             Objects.equals(name, other.name) &&
             Objects.equals(schedule, other.schedule) &&
             Objects.equals(repository, other.repository) &&
-            Objects.equals(configuration, other.configuration);
+            Objects.equals(configuration, other.configuration) &&
+            Objects.equals(retentionPolicy, other.retentionPolicy);
     }
 
     @Override

client/rest-high-level/src/main/java/org/elasticsearch/client/slm/SnapshotLifecyclePolicyMetadata.java

@@ -42,6 +42,7 @@ public class SnapshotLifecyclePolicyMetadata implements ToXContentObject {
     static final ParseField NEXT_EXECUTION_MILLIS = new ParseField("next_execution_millis");
     static final ParseField NEXT_EXECUTION = new ParseField("next_execution");
     static final ParseField SNAPSHOT_IN_PROGRESS = new ParseField("in_progress");
+    static final ParseField POLICY_STATS = new ParseField("stats");
 
     private final SnapshotLifecyclePolicy policy;
     private final long version;
@@ -53,6 +54,7 @@ public class SnapshotLifecyclePolicyMetadata implements ToXContentObject {
     private final SnapshotInvocationRecord lastFailure;
     @Nullable
     private final SnapshotInProgress snapshotInProgress;
+    private final SnapshotLifecycleStats.SnapshotPolicyStats policyStats;
 
     @SuppressWarnings("unchecked")
     public static final ConstructingObjectParser<SnapshotLifecyclePolicyMetadata, String> PARSER =
@@ -65,8 +67,9 @@ public class SnapshotLifecyclePolicyMetadata implements ToXContentObject {
                 SnapshotInvocationRecord lastFailure = (SnapshotInvocationRecord) a[4];
                 long nextExecution = (long) a[5];
                 SnapshotInProgress sip = (SnapshotInProgress) a[6];
-
-                return new SnapshotLifecyclePolicyMetadata(policy, version, modifiedDate, lastSuccess, lastFailure, nextExecution, sip);
+                SnapshotLifecycleStats.SnapshotPolicyStats stats = (SnapshotLifecycleStats.SnapshotPolicyStats) a[7];
+                return new SnapshotLifecyclePolicyMetadata(policy, version, modifiedDate, lastSuccess,
+                    lastFailure, nextExecution, sip, stats);
             });
 
     static {
@@ -77,6 +80,9 @@ public class SnapshotLifecyclePolicyMetadata implements ToXContentObject {
         PARSER.declareObject(ConstructingObjectParser.optionalConstructorArg(), SnapshotInvocationRecord::parse, LAST_FAILURE);
         PARSER.declareLong(ConstructingObjectParser.constructorArg(), NEXT_EXECUTION_MILLIS);
         PARSER.declareObject(ConstructingObjectParser.optionalConstructorArg(), SnapshotInProgress::parse, SNAPSHOT_IN_PROGRESS);
+        PARSER.declareObject(ConstructingObjectParser.constructorArg(),
+            (p, c) -> SnapshotLifecycleStats.SnapshotPolicyStats.parse(p, "policy"), POLICY_STATS);
+
     }
 
     public static SnapshotLifecyclePolicyMetadata parse(XContentParser parser, String id) {
@@ -86,14 +92,16 @@ public static SnapshotLifecyclePolicyMetadata parse(XContentParser parser, Strin
     public SnapshotLifecyclePolicyMetadata(SnapshotLifecyclePolicy policy, long version, long modifiedDate,
                                            SnapshotInvocationRecord lastSuccess, SnapshotInvocationRecord lastFailure,
                                            long nextExecution,
-                                           @Nullable SnapshotInProgress snapshotInProgress) {
+                                           @Nullable SnapshotInProgress snapshotInProgress,
+                                           SnapshotLifecycleStats.SnapshotPolicyStats policyStats) {
         this.policy = policy;
         this.version = version;
         this.modifiedDate = modifiedDate;
         this.lastSuccess = lastSuccess;
         this.lastFailure = lastFailure;
         this.nextExecution = nextExecution;
         this.snapshotInProgress = snapshotInProgress;
+        this.policyStats = policyStats;
     }
 
     public SnapshotLifecyclePolicy getPolicy() {
@@ -124,6 +132,10 @@ public long getNextExecution() {
         return this.nextExecution;
     }
 
+    public SnapshotLifecycleStats.SnapshotPolicyStats getPolicyStats() {
+        return this.policyStats;
+    }
+
     @Nullable
     public SnapshotInProgress getSnapshotInProgress() {
         return this.snapshotInProgress;
@@ -145,13 +157,16 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
         if (snapshotInProgress != null) {
             builder.field(SNAPSHOT_IN_PROGRESS.getPreferredName(), snapshotInProgress);
         }
+        builder.startObject(POLICY_STATS.getPreferredName());
+        this.policyStats.toXContent(builder, params);
+        builder.endObject();
         builder.endObject();
         return builder;
     }
 
     @Override
     public int hashCode() {
-        return Objects.hash(policy, version, modifiedDate, lastSuccess, lastFailure, nextExecution);
+        return Objects.hash(policy, version, modifiedDate, lastSuccess, lastFailure, nextExecution, policyStats);
     }
 
     @Override
@@ -168,7 +183,8 @@ public boolean equals(Object obj) {
             Objects.equals(modifiedDate, other.modifiedDate) &&
             Objects.equals(lastSuccess, other.lastSuccess) &&
             Objects.equals(lastFailure, other.lastFailure) &&
-            Objects.equals(nextExecution, other.nextExecution);
+            Objects.equals(nextExecution, other.nextExecution) &&
+            Objects.equals(policyStats, other.policyStats);
     }
 
     public static class SnapshotInProgress implements ToXContentObject {