Merge remote-tracking branch 'elastic/master' into thread-context-add-response-header-performance

* elastic/master:
  Remove Redundant RestoreRequest Class (elastic#37535)
  Create specific exception for when snapshots are in progress (elastic#37550)
  Mute UnicastZenPingTests#testSimplePings
  [DOCS] Adds size limitation to the get datafeeds APIs (elastic#37578)
  Fix assertion at end of forceRefreshes (elastic#37559)
  Propagate Errors in executors to uncaught exception handler (elastic#36137)
  [DOCS] Adds limitation to the get jobs API (elastic#37549)
  Add set_priority action to ILM (elastic#37397)
  Make recovery source send operations non-blocking (elastic#37503)
  Allow field types to optimize phrase prefix queries (elastic#37436)
  Added fatal_exception field for ccr stats in monitoring mapping. (elastic#37563)
  Fix testRelocateWhileContinuouslyIndexingAndWaitingForRefresh (elastic#37560)
  Moved ccr integration to the package with other ccr integration tests.
  Mute TransportClientNodesServiceTests#testListenerFailures
  Decreased time out in test
  Fix erroneous docstrings for abstract bulk by scroll request (elastic#37517)
  SQL: Rename SQL type DATE to DATETIME (elastic#37395)
  Remove the AbstracLifecycleComponent constructor with Settings (elastic#37523)

Author: jasontedor

buildSrc/src/main/resources/forbidden/es-all-signatures.txt

@@ -50,3 +50,16 @@ java.nio.channels.SocketChannel#connect(java.net.SocketAddress)
 java.lang.Boolean#getBoolean(java.lang.String)
 
 org.apache.lucene.util.IOUtils @ use @org.elasticsearch.core.internal.io instead
+
+@defaultMessage use executors from org.elasticsearch.common.util.concurrent.EsExecutors instead which will properly bubble up Errors
+java.util.concurrent.AbstractExecutorService#<init>()
+java.util.concurrent.ThreadPoolExecutor#<init>(int, int, long, java.util.concurrent.TimeUnit, java.util.concurrent.BlockingQueue)
+java.util.concurrent.ThreadPoolExecutor#<init>(int, int, long, java.util.concurrent.TimeUnit, java.util.concurrent.BlockingQueue, java.util.concurrent.ThreadFactory)
+java.util.concurrent.ThreadPoolExecutor#<init>(int, int, long, java.util.concurrent.TimeUnit, java.util.concurrent.BlockingQueue, java.util.concurrent.RejectedExecutionHandler)
+java.util.concurrent.ThreadPoolExecutor#<init>(int, int, long, java.util.concurrent.TimeUnit, java.util.concurrent.BlockingQueue, java.util.concurrent.ThreadFactory, java.util.concurrent.RejectedExecutionHandler)
+
+@defaultMessage extend org.elasticsearch.threadpool.Scheduler.SafeScheduledThreadPoolExecutor instead which will properly bubble up Errors
+java.util.concurrent.ScheduledThreadPoolExecutor#<init>(int)
+java.util.concurrent.ScheduledThreadPoolExecutor#<init>(int, java.util.concurrent.ThreadFactory)
+java.util.concurrent.ScheduledThreadPoolExecutor#<init>(int, java.util.concurrent.RejectedExecutionHandler)
+java.util.concurrent.ScheduledThreadPoolExecutor#<init>(int, java.util.concurrent.ThreadFactory, java.util.concurrent.RejectedExecutionHandler)

client/rest-high-level/src/main/java/org/elasticsearch/client/indexlifecycle/IndexLifecycleNamedXContentProvider.java

@@ -53,7 +53,10 @@ public List<NamedXContentRegistry.Entry> getNamedXContentParsers() {
                 ShrinkAction::parse),
             new NamedXContentRegistry.Entry(LifecycleAction.class,
                 new ParseField(FreezeAction.NAME),
-                FreezeAction::parse)
+                FreezeAction::parse),
+            new NamedXContentRegistry.Entry(LifecycleAction.class,
+                new ParseField(SetPriorityAction.NAME),
+                SetPriorityAction::parse)
         );
     }
 }

client/rest-high-level/src/main/java/org/elasticsearch/client/indexlifecycle/LifecyclePolicy.java

@@ -57,9 +57,10 @@ public class LifecyclePolicy implements ToXContentObject {
             throw new IllegalArgumentException("ordered " + PHASES_FIELD.getPreferredName() + " are not supported");
         }, PHASES_FIELD);
 
-        ALLOWED_ACTIONS.put("hot", Sets.newHashSet(RolloverAction.NAME));
-        ALLOWED_ACTIONS.put("warm", Sets.newHashSet(AllocateAction.NAME, ForceMergeAction.NAME, ReadOnlyAction.NAME, ShrinkAction.NAME));
-        ALLOWED_ACTIONS.put("cold", Sets.newHashSet(AllocateAction.NAME, FreezeAction.NAME));
+        ALLOWED_ACTIONS.put("hot", Sets.newHashSet(SetPriorityAction.NAME, RolloverAction.NAME));
+        ALLOWED_ACTIONS.put("warm", Sets.newHashSet(SetPriorityAction.NAME, AllocateAction.NAME, ForceMergeAction.NAME,
+            ReadOnlyAction.NAME, ShrinkAction.NAME));
+        ALLOWED_ACTIONS.put("cold", Sets.newHashSet(SetPriorityAction.NAME, AllocateAction.NAME, FreezeAction.NAME));
         ALLOWED_ACTIONS.put("delete", Sets.newHashSet(DeleteAction.NAME));
     }
 

client/rest-high-level/src/main/java/org/elasticsearch/client/indexlifecycle/SetPriorityAction.java

@@ -0,0 +1,96 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.client.indexlifecycle;
+
+import org.elasticsearch.common.Nullable;
+import org.elasticsearch.common.ParseField;
+import org.elasticsearch.common.Strings;
+import org.elasticsearch.common.xcontent.ConstructingObjectParser;
+import org.elasticsearch.common.xcontent.ObjectParser;
+import org.elasticsearch.common.xcontent.ToXContent;
+import org.elasticsearch.common.xcontent.ToXContentObject;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.common.xcontent.XContentParser;
+
+import java.io.IOException;
+
+/**
+ * A {@link LifecycleAction} which sets the index's priority. The higher the priority, the faster the recovery.
+ */
+public class SetPriorityAction implements LifecycleAction, ToXContentObject {
+    public static final String NAME = "set_priority";
+    private static final ParseField RECOVERY_PRIORITY_FIELD = new ParseField("priority");
+
+    @SuppressWarnings("unchecked")
+    private static final ConstructingObjectParser<SetPriorityAction, Void> PARSER = new ConstructingObjectParser<>(NAME, true,
+        a -> new SetPriorityAction((Integer) a[0]));
+
+    //package private for testing
+    final Integer recoveryPriority;
+
+    static {
+        PARSER.declareField(ConstructingObjectParser.constructorArg(),
+            (p) -> p.currentToken() == XContentParser.Token.VALUE_NULL ? null : p.intValue()
+            , RECOVERY_PRIORITY_FIELD, ObjectParser.ValueType.INT_OR_NULL);
+    }
+
+    public static SetPriorityAction parse(XContentParser parser) {
+        return PARSER.apply(parser, null);
+    }
+
+    public SetPriorityAction(@Nullable Integer recoveryPriority) {
+        if (recoveryPriority != null && recoveryPriority <= 0) {
+            throw new IllegalArgumentException("[" + RECOVERY_PRIORITY_FIELD.getPreferredName() + "] must be 0 or greater");
+        }
+        this.recoveryPriority = recoveryPriority;
+    }
+
+    @Override
+    public XContentBuilder toXContent(XContentBuilder builder, ToXContent.Params params) throws IOException {
+        builder.startObject();
+        builder.field(RECOVERY_PRIORITY_FIELD.getPreferredName(), recoveryPriority);
+        builder.endObject();
+        return builder;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+
+        SetPriorityAction that = (SetPriorityAction) o;
+
+        return recoveryPriority != null ? recoveryPriority.equals(that.recoveryPriority) : that.recoveryPriority == null;
+    }
+
+    @Override
+    public int hashCode() {
+        return recoveryPriority != null ? recoveryPriority.hashCode() : 0;
+    }
+
+    @Override
+    public String toString() {
+        return Strings.toString(this);
+    }
+
+    @Override
+    public String getName() {
+        return NAME;
+    }
+}

client/rest-high-level/src/test/java/org/elasticsearch/client/RestHighLevelClientTests.java

@@ -54,6 +54,7 @@
 import org.elasticsearch.client.indexlifecycle.LifecycleAction;
 import org.elasticsearch.client.indexlifecycle.ReadOnlyAction;
 import org.elasticsearch.client.indexlifecycle.RolloverAction;
+import org.elasticsearch.client.indexlifecycle.SetPriorityAction;
 import org.elasticsearch.client.indexlifecycle.ShrinkAction;
 import org.elasticsearch.cluster.ClusterName;
 import org.elasticsearch.common.CheckedFunction;
@@ -644,7 +645,7 @@ public void testDefaultNamedXContents() {
 
     public void testProvidedNamedXContents() {
         List<NamedXContentRegistry.Entry> namedXContents = RestHighLevelClient.getProvidedNamedXContents();
-        assertEquals(18, namedXContents.size());
+        assertEquals(19, namedXContents.size());
         Map<Class<?>, Integer> categories = new HashMap<>();
         List<String> names = new ArrayList<>();
         for (NamedXContentRegistry.Entry namedXContent : namedXContents) {
@@ -668,14 +669,15 @@ public void testProvidedNamedXContents() {
         assertTrue(names.contains(MeanReciprocalRank.NAME));
         assertTrue(names.contains(DiscountedCumulativeGain.NAME));
         assertTrue(names.contains(ExpectedReciprocalRank.NAME));
-        assertEquals(Integer.valueOf(7), categories.get(LifecycleAction.class));
+        assertEquals(Integer.valueOf(8), categories.get(LifecycleAction.class));
         assertTrue(names.contains(AllocateAction.NAME));
         assertTrue(names.contains(DeleteAction.NAME));
         assertTrue(names.contains(ForceMergeAction.NAME));
         assertTrue(names.contains(ReadOnlyAction.NAME));
         assertTrue(names.contains(RolloverAction.NAME));
         assertTrue(names.contains(ShrinkAction.NAME));
         assertTrue(names.contains(FreezeAction.NAME));
+        assertTrue(names.contains(SetPriorityAction.NAME));
     }
 
     public void testApiNamingConventions() throws Exception {

client/rest-high-level/src/test/java/org/elasticsearch/client/indexlifecycle/GetLifecyclePolicyResponseTests.java

@@ -67,7 +67,8 @@ protected NamedXContentRegistry xContentRegistry() {
             new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(ReadOnlyAction.NAME), ReadOnlyAction::parse),
             new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(RolloverAction.NAME), RolloverAction::parse),
             new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(ShrinkAction.NAME), ShrinkAction::parse),
-            new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(FreezeAction.NAME), FreezeAction::parse)
+            new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(FreezeAction.NAME), FreezeAction::parse),
+            new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(SetPriorityAction.NAME), SetPriorityAction::parse)
         ));
         return new NamedXContentRegistry(entries);
     }

client/rest-high-level/src/test/java/org/elasticsearch/client/indexlifecycle/LifecyclePolicyMetadataTests.java

@@ -63,7 +63,8 @@ protected NamedXContentRegistry xContentRegistry() {
             new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(ReadOnlyAction.NAME), ReadOnlyAction::parse),
             new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(RolloverAction.NAME), RolloverAction::parse),
             new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(ShrinkAction.NAME), ShrinkAction::parse),
-            new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(FreezeAction.NAME), FreezeAction::parse)
+            new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(FreezeAction.NAME), FreezeAction::parse),
+            new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(SetPriorityAction.NAME), SetPriorityAction::parse)
         ));
         return new NamedXContentRegistry(entries);
     }

client/rest-high-level/src/test/java/org/elasticsearch/client/indexlifecycle/LifecyclePolicyTests.java

@@ -39,10 +39,10 @@
 import static org.hamcrest.Matchers.equalTo;
 
 public class LifecyclePolicyTests extends AbstractXContentTestCase<LifecyclePolicy> {
-    private static final Set<String> VALID_HOT_ACTIONS = Sets.newHashSet(RolloverAction.NAME);
-    private static final Set<String> VALID_WARM_ACTIONS = Sets.newHashSet(AllocateAction.NAME, ForceMergeAction.NAME,
-        ReadOnlyAction.NAME, ShrinkAction.NAME);
-    private static final Set<String> VALID_COLD_ACTIONS = Sets.newHashSet(AllocateAction.NAME, FreezeAction.NAME);
+    private static final Set<String> VALID_HOT_ACTIONS = Sets.newHashSet(SetPriorityAction.NAME, RolloverAction.NAME);
+    private static final Set<String> VALID_WARM_ACTIONS = Sets.newHashSet(SetPriorityAction.NAME, AllocateAction.NAME,
+        ForceMergeAction.NAME, ReadOnlyAction.NAME, ShrinkAction.NAME);
+    private static final Set<String> VALID_COLD_ACTIONS = Sets.newHashSet(SetPriorityAction.NAME, AllocateAction.NAME, FreezeAction.NAME);
     private static final Set<String> VALID_DELETE_ACTIONS = Sets.newHashSet(DeleteAction.NAME);
 
     private String lifecycleName;
@@ -67,7 +67,8 @@ protected NamedXContentRegistry xContentRegistry() {
             new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(ReadOnlyAction.NAME), ReadOnlyAction::parse),
             new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(RolloverAction.NAME), RolloverAction::parse),
             new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(ShrinkAction.NAME), ShrinkAction::parse),
-            new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(FreezeAction.NAME), FreezeAction::parse)
+            new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(FreezeAction.NAME), FreezeAction::parse),
+            new NamedXContentRegistry.Entry(LifecycleAction.class, new ParseField(SetPriorityAction.NAME), SetPriorityAction::parse)
         ));
         return new NamedXContentRegistry(entries);
     }
@@ -210,6 +211,8 @@ public static LifecyclePolicy createRandomPolicy(String lifecycleName) {
                     return ShrinkActionTests.randomInstance();
                 case FreezeAction.NAME:
                     return new FreezeAction();
+                case SetPriorityAction.NAME:
+                    return SetPriorityActionTests.randomInstance();
                 default:
                     throw new IllegalArgumentException("invalid action [" + action + "]");
             }};
@@ -241,6 +244,8 @@ private LifecycleAction getTestAction(String actionName) {
                 return ShrinkActionTests.randomInstance();
             case FreezeAction.NAME:
                 return new FreezeAction();
+            case SetPriorityAction.NAME:
+                return SetPriorityActionTests.randomInstance();
             default:
                 throw new IllegalArgumentException("unsupported phase action [" + actionName + "]");
         }

client/rest-high-level/src/test/java/org/elasticsearch/client/indexlifecycle/SetPriorityActionTests.java

@@ -0,0 +1,71 @@
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.client.indexlifecycle;
+
+import org.elasticsearch.common.xcontent.XContentParser;
+import org.elasticsearch.test.AbstractXContentTestCase;
+import org.elasticsearch.test.EqualsHashCodeTestUtils;
+
+import java.io.IOException;
+
+import static org.hamcrest.Matchers.equalTo;
+
+public class SetPriorityActionTests extends AbstractXContentTestCase<SetPriorityAction> {
+
+    @Override
+    protected SetPriorityAction doParseInstance(XContentParser parser) throws IOException {
+        return SetPriorityAction.parse(parser);
+    }
+
+    @Override
+    protected SetPriorityAction createTestInstance() {
+        return randomInstance();
+    }
+
+    static SetPriorityAction randomInstance() {
+        return new SetPriorityAction(randomIntBetween(1, 100));
+    }
+
+    @Override
+    protected boolean supportsUnknownFields() {
+        return false;
+    }
+
+    public void testNonPositivePriority() {
+        Exception e = expectThrows(Exception.class, () -> new SetPriorityAction(randomIntBetween(-100, 0)));
+        assertThat(e.getMessage(), equalTo("[priority] must be 0 or greater"));
+    }
+
+    public void testNullPriorityAllowed(){
+        SetPriorityAction nullPriority = new SetPriorityAction(null);
+        assertNull(nullPriority.recoveryPriority);
+    }
+
+    public void testEqualsAndHashCode() {
+        EqualsHashCodeTestUtils.checkEqualsAndHashCode(createTestInstance(), this::copy);
+    }
+
+    SetPriorityAction copy(SetPriorityAction setPriorityAction) {
+        return new SetPriorityAction(setPriorityAction.recoveryPriority);
+    }
+
+    SetPriorityAction notCopy(SetPriorityAction setPriorityAction) {
+        return new SetPriorityAction(setPriorityAction.recoveryPriority + 1);
+    }
+}

docs/reference/ilm/policy-definitions.asciidoc

@@ -85,13 +85,16 @@ executing.
 The below list shows the actions which are available in each phase.
 
 * Hot
+  - <<ilm-set-priority-action,Set Priority>>
   - <<ilm-rollover-action,Rollover>>
 * Warm
+  - <<ilm-set-priority-action,Set Priority>>
   - <<ilm-allocate-action,Allocate>>
   - <<ilm-readonly-action,Read-Only>>
   - <<ilm-forcemerge-action,Force Merge>>
   - <<ilm-shrink-action,Shrink>>
 * Cold
+  - <<ilm-set-priority-action,Set Priority>>
   - <<ilm-allocate-action,Allocate>>
   - <<ilm-freeze-action,Freeze>>
 * Delete
@@ -525,6 +528,48 @@ The above example illustrates a policy which attempts to delete an
 index one day after the index has been rolled over. It does not
 delete the index one day after it has been created.
 
+[[ilm-set-priority-action]]
+==== Set Priority
+
+Phases allowed: hot, warm, cold.
+
+This action sets the <<recovery-prioritization, index priority>> on the index as
+soon as the policy enters the hot, warm, or cold phase. Indices with a higher
+priority will be recovered before indices with lower priorities following a node
+restart. Generally, indexes in the hot phase should have the highest value and
+indexes in the cold phase should have the lowest values. For example:
+100 for the hot phase, 50 for the warm phase, and 0 for the cold phase.
+Indicies that don't set this value have an implicit default priority of 1.
+
+[[ilm-set-priority-options]]
+.Set Priority Options
+[options="header"]
+|======
+| Name         | Required  | Default     | Description
+| `priority`   | yes       | -           | The priority for the index. Must be 0 or greater.
+                                           The value may also be set to null to remove the priority.
+
+|======
+
+[source,js]
+--------------------------------------------------
+PUT _ilm/policy/my_policy
+{
+  "policy": {
+    "phases": {
+      "warm": {
+        "actions": {
+          "set_priority" : {
+            "priority": 50
+          }
+        }
+      }
+    }
+  }
+}
+--------------------------------------------------
+// CONSOLE
+
 [[ilm-shrink-action]]
 ==== Shrink
 

docs/reference/migration/migrate_7_0/java.asciidoc

@@ -38,4 +38,10 @@ because `Settings` is no longer needed.
 ==== Deprecated method `Client#termVector` removed
 
 The client method `termVector`, deprecated in 2.0, has been removed. The method
-`termVectors` (plural) should be used instead.
+`termVectors` (plural) should be used instead.
+
+[float]
+==== Deprecated constructor `AbstractLifecycleComponent(Settings settings)` removed
+
+The constructor `AbstractLifecycleComponent(Settings settings)`, deprecated in 6.7
+has been removed. The parameterless constructor should be used instead.

docs/reference/ml/apis/get-datafeed-stats.asciidoc

@@ -32,6 +32,7 @@ statistics for all {dfeeds} by using `_all`, by specifying `*` as the
 If the {dfeed} is stopped, the only information you receive is the
 `datafeed_id` and the `state`.
 
+IMPORTANT: This API returns a maximum of 10,000 {dfeeds}.
 
 ==== Path Parameters