Merge remote-tracking branch 'elastic/master' into license-to-autoscale

* elastic/master: (33 commits)
  Add searchable snapshot cache folder to NodeEnvironment (elastic#66297)
  [DOCS] Add dynamic runtime fields to docs (elastic#66194)
  Add HDFS searchable snapshot integration (elastic#66185)
  Support canceling cross-clusters search requests (elastic#66206)
  Mute testCacheSurviveRestart (elastic#66289)
  Fix cat tasks api params in spec and handler (elastic#66272)
  Snapshot of a searchable snapshot should be empty (elastic#66162)
  [ML] DFA _explain API should not fail when none field is included (elastic#66281)
  Add action to decommission legacy monitoring cluster alerts (elastic#64373)
  move rollup_index param out of RollupActionConfig (elastic#66139)
  Improve FieldFetcher retrieval of fields (elastic#66160)
  Remove unsed fields in `RestAnalyzeAction` (elastic#66215)
  Simplify searchable snapshot CacheKey (elastic#66263)
  Autoscaling remove feature flags (elastic#65973)
  Improve searchable snapshot mount time (elastic#66198)
  [ML] Report cause when datafeed extraction encounters error (elastic#66167)
  Remove suggest reference in some API specs (elastic#66180)
  Fix warning when installing a plugin for different ESversion (elastic#66146)
  [ML] make `xpack.ml.max_ml_node_size` and `xpack.ml.use_auto_machine_memory_percent` dynamically settable (elastic#66132)
  [DOCS] Add `require_alias` to Bulk API (elastic#66259)
  ...

Author: jasontedor

.editorconfig

@@ -13,9 +13,11 @@ indent_size = 2
 
 [*.groovy]
 indent_size = 4
+max_line_length = 140
 
 [*.java]
 indent_size = 4
+max_line_length = 140
 
 [*.json]
 indent_size = 2

CONTRIBUTING.md

@@ -164,6 +164,47 @@ You can import the Elasticsearch project into IntelliJ IDEA via:
  - In the subsequent dialog navigate to the root `build.gradle` file
  - In the subsequent dialog select **Open as Project**
 
+#### Checkstyle
+
+If you have the [Checkstyle] plugin installed, you can configure IntelliJ to
+check the Elasticsearch code. However, the Checkstyle configuration file does
+not work by default with the IntelliJ plugin, so instead an IDE-specific config
+file is generated automatically after IntelliJ finishes syncing.
+
+   1. Open **Preferences > Tools > Checkstyle**
+   2. Change the "Scan Scope" to "Only Java sources (including tests)"
+   3. Check the "+" under "Configuration file"
+   4. Set "Description" to "Elasticsearch" (or whatever you want)
+   5. Select "Use a local Checkstyle file"
+   6. For the "File", navigate to `build/checkstyle_ide.xml`
+   7. Tick "Store relative to project location"
+   8. Click "Next"
+   9. The Checkstyle config file contains the variable `config_loc`, and
+      IntelliJ will ask for a value. Fill in `buildSrc/src/main/resources`.
+      This allows the config file to reference the exclusions file in that directory.
+   10. Click "Next", then "Finish".
+   11. Click the box next to the new configuration to make it "Active". Without doing this,
+       you'll have to explicitly choose the "Elasticsearch" configuration in the Checkstyle
+       tool window and run the check manually. You can still do this with an active config.
+   12. Click "OK" to apply the new preferences
+
+#### Formatting
+
+We are in the process of migrating towards automatic formatting Java file
+using [spotless], backed by the Eclipse formatter. If you have the [Eclipse
+Code Formatter] installed, you can apply formatting directly in IntelliJ.
+
+   1. Open **Preferences > Other Settings > Eclipse Code Formatter**
+   2. Click "Use the Eclipse Code Formatter"
+   3. Under "Eclipse formatter config", select "Eclipse workspace/project
+      folder or config file"
+   4. Click "Browse", and navigate to the file `buildSrc/formatterConfig.xml`
+   5. Click "OK"
+
+Note that only some sub-projects in the Elasticsearch project are currently
+fully-formatted. You can see a list of project that **are not**
+automatically formatted in [gradle/formatting.gradle](gradle/formatting.gradle).
+
 ### Importing the project into Eclipse
 
 Elasticsearch builds using Gradle and Java 14. When importing into Eclipse you
@@ -699,3 +740,6 @@ non-documentation contribution. This is mentioned above, but it is worth
 repeating in this section because it has come up in this context.
 
 [intellij]: https://blog.jetbrains.com/idea/2017/07/intellij-idea-2017-2-is-here-smart-sleek-and-snappy/
+[Checkstyle]: https://plugins.jetbrains.com/plugin/1065-checkstyle-idea
+[spotless]: https://github.com/diffplug/spotless
+[Eclipse Code Formatter]: https://plugins.jetbrains.com/plugin/6546-eclipse-code-formatter

client/rest-high-level/src/main/java/org/elasticsearch/client/ml/job/config/CategorizationAnalyzerConfig.java

@@ -18,6 +18,7 @@
  */
 package org.elasticsearch.client.ml.job.config;
 
+import org.elasticsearch.action.admin.indices.analyze.AnalyzeAction;
 import org.elasticsearch.common.ParseField;
 import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.ToXContentFragment;
@@ -59,9 +60,9 @@
 public class CategorizationAnalyzerConfig implements ToXContentFragment {
 
     public static final ParseField CATEGORIZATION_ANALYZER = new ParseField("categorization_analyzer");
-    private static final ParseField TOKENIZER = RestAnalyzeAction.Fields.TOKENIZER;
-    private static final ParseField TOKEN_FILTERS = RestAnalyzeAction.Fields.TOKEN_FILTERS;
-    private static final ParseField CHAR_FILTERS = RestAnalyzeAction.Fields.CHAR_FILTERS;
+    private static final ParseField TOKENIZER = AnalyzeAction.Fields.TOKENIZER;
+    private static final ParseField TOKEN_FILTERS = AnalyzeAction.Fields.TOKEN_FILTERS;
+    private static final ParseField CHAR_FILTERS = AnalyzeAction.Fields.CHAR_FILTERS;
 
     /**
      * This method is only used in the unit tests - in production code this config is always parsed as a fragment.

distribution/tools/plugin-cli/src/main/java/org/elasticsearch/plugins/ListPluginsCommand.java

@@ -70,7 +70,7 @@ private void printPlugin(Environment env, Terminal terminal, Path plugin, String
                 "WARNING: plugin ["
                     + info.getName()
                     + "] was built for Elasticsearch version "
-                    + info.getVersion()
+                    + info.getElasticsearchVersion()
                     + " but version "
                     + Version.CURRENT
                     + " is required"

distribution/tools/plugin-cli/src/test/java/org/elasticsearch/plugins/ListPluginsCommandTests.java

@@ -266,7 +266,7 @@ public void testExistingIncompatiblePlugin() throws Exception {
         buildFakePlugin(env, "fake desc 2", "fake_plugin2", "org.fake2");
 
         MockTerminal terminal = listPlugins(home);
-        String message = "plugin [fake_plugin1] was built for Elasticsearch version 1.0 but version " + Version.CURRENT + " is required";
+        String message = "plugin [fake_plugin1] was built for Elasticsearch version 1.0.0 but version " + Version.CURRENT + " is required";
         assertEquals("fake_plugin1\nfake_plugin2\n", terminal.getOutput());
         assertEquals("WARNING: " + message + "\n", terminal.getErrorOutput());
 

docs/build.gradle

@@ -54,10 +54,6 @@ testClusters.matching { it.name == "integTest"}.configureEach {
   if (singleNode().testDistribution == DEFAULT) {
     setting 'xpack.license.self_generated.type', 'trial'
     setting 'indices.lifecycle.history_index_enabled', 'false'
-    if (BuildParams.isSnapshotBuild() == false) {
-      systemProperty 'es.autoscaling_feature_flag_registered', 'true'
-    }
-    setting 'xpack.autoscaling.enabled', 'true'
     systemProperty 'es.rollup_v2_feature_flag_enabled', 'true'
     keystorePassword 's3cr3t'
   }

docs/reference/autoscaling/apis/autoscaling-apis.asciidoc

@@ -3,6 +3,8 @@
 [[autoscaling-apis]]
 == Autoscaling APIs
 
+experimental[]
+
 You can use the following APIs to perform autoscaling operations.
 
 [discrete]

docs/reference/autoscaling/apis/delete-autoscaling-policy.asciidoc

@@ -6,6 +6,8 @@
 <titleabbrev>Delete autoscaling policy</titleabbrev>
 ++++
 
+experimental[]
+
 Delete autoscaling policy.
 
 [[autoscaling-delete-autoscaling-policy-request]]

docs/reference/autoscaling/apis/get-autoscaling-capacity.asciidoc

@@ -6,6 +6,8 @@
 <titleabbrev>Get autoscaling capacity</titleabbrev>
 ++++
 
+experimental[]
+
 Get autoscaling capacity.
 
 [[autoscaling-get-autoscaling-capacity-request]]

docs/reference/autoscaling/apis/get-autoscaling-policy.asciidoc

@@ -6,6 +6,8 @@
 <titleabbrev>Get autoscaling policy</titleabbrev>
 ++++
 
+experimental[]
+
 Get autoscaling policy.
 
 [[autoscaling-get-autoscaling-policy-request]]

docs/reference/autoscaling/apis/put-autoscaling-policy.asciidoc

@@ -6,6 +6,8 @@
 <titleabbrev>Put autoscaling policy</titleabbrev>
 ++++
 
+experimental[]
+
 Put autoscaling policy.
 
 [[autoscaling-put-autoscaling-policy-request]]

docs/reference/docs/bulk.asciidoc

@@ -211,6 +211,11 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=pipeline]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=refresh]
 
+`require_alias`::
+(Optional, Boolean)
+If `true`, the request's actions must target an <<indices-aliases,index alias>>.
+Defaults to `false`.
+
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=routing]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source]
@@ -237,6 +242,8 @@ The following line must contain the source data to be indexed.
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-index]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-id]
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-require-alias]
 --
 
 `delete`::
@@ -249,6 +256,8 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-index]
 `_id`::
 (Required, string) 
 The document ID.
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-require-alias]
 --
 
 `index`::
@@ -261,6 +270,8 @@ The following line must contain the source data to be indexed.
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-index]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-id]
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-require-alias]
 --
 
 `update`::
@@ -272,6 +283,8 @@ The following line must contain the partial document and update options.
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-index]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-id]
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bulk-require-alias]
 --
 
 `doc`::

docs/reference/docs/data-replication.asciidoc

@@ -108,7 +108,8 @@ is as follows:
 . Resolve the read requests to the relevant shards. Note that since most searches will be sent to one or more indices,
    they typically need to read from multiple shards, each representing a different subset of the data.
 . Select an active copy of each relevant shard, from the shard replication group. This can be either the primary or
-   a replica. By default, Elasticsearch will simply round robin between the shard copies.
+   a replica. By default, {es} uses <<search-adaptive-replica,adaptive replica
+   selection>> to select the shard copies.
 . Send shard level read requests to the selected copies.
 . Combine the results and respond. Note that in the case of get by ID look up, only one shard is relevant and this step can be skipped.
 

docs/reference/index.asciidoc

@@ -48,12 +48,8 @@ include::data-management.asciidoc[]
 
 include::ilm/index.asciidoc[]
 
-ifdef::permanently-unreleased-branch[]
-
 include::autoscaling/index.asciidoc[]
 
-endif::[]
-
 include::monitoring/index.asciidoc[]
 
 include::frozen-indices.asciidoc[]

docs/reference/mapping/dynamic/field-mapping.asciidoc

@@ -1,35 +1,37 @@
 [[dynamic-field-mapping]]
 === Dynamic field mapping
 
-By default, when a previously unseen field is found in a document,
-Elasticsearch will add the new field to the type mapping.   This behaviour can
-be disabled, both at the document and at the <<object,`object`>>  level, by
-setting the <<dynamic,`dynamic`>> parameter to `false` (to ignore new fields) or to `strict` (to throw
-an exception if an unknown field is encountered).
-
-Assuming `dynamic` field mapping is enabled, some simple rules are used to
-determine which data type the field should have:
-
-[horizontal]
-*JSON data type*::                   *Elasticsearch data type*
-
-`null`::                            No field is added.
-`true` or `false`::                 <<boolean,`boolean`>> field
-floating{nbsp}point{nbsp}number::   <<number,`float`>> field
-integer::                           <<number,`long`>> field
-object::                            <<object,`object`>> field
-array::                             Depends on the first non-`null` value in the array.
-string::                            Either a <<date,`date`>> field
-                                        (if the value passes <<date-detection,date detection>>),
-                                    a <<number,`double`>> or <<number,`long`>> field
-                                        (if the value passes <<numeric-detection,numeric detection>>)
-                                    or a <<text,`text`>> field, with a <<keyword,`keyword`>> sub-field.
-
-These are the only <<mapping-types,field data types>> that are dynamically
-detected.  All other data types must be mapped explicitly.
-
-Besides the options listed below, dynamic field mapping rules can be further
-customised with <<dynamic-templates,`dynamic_templates`>>.
+When {es} detects a new field in a document, it _dynamically_ adds the field to
+the type mapping. The <<dynamic,`dynamic`>> parameter controls this behavior.
+
+You can disable dynamic mapping, both at the document and at the
+<<object,`object`>> level. Setting the `dynamic` parameter to
+`false` ignores new fields, and `strict` rejects the document if {es}
+encounters an unknown field.
+
+When dynamic field mapping is enabled, {es} uses the following rules to
+determine how to map data types for each field.
+
+NOTE: These are the only <<mapping-types,field data types>> that {es} detects
+dynamically. All other data types must be mapped explicitly.
+
+[cols="3"]
+|===
+h| JSON data type h| `dynamic:true` h| `dynamic:runtime`
+ |`null` 2*| No field added
+ |`true` or `false` 2*| `boolean`
+ |floating point number | `float` | `double`
+ |`integer` 2*| `long`
+ |`object`<<dynamic-object-footnote,^1^>>  2*| `object`
+ |`array` 2*|  Depends on the first non-`null` value in the array
+ |`string` that passes <<date-detection,date detection>> 2*| `date`
+ |`string` that passes <<numeric-detection,numeric detection>> | `double` or `long` | `double`
+ |`string` that doesn't pass `date` detection or `numeric` detection | `text` with a `.keyword` sub-field | `keyword`
+3+| [[dynamic-object-footnote]] ^1^Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | |
+|===
+
+Besides the following options, you can customize dynamic field mapping rules
+with <<dynamic-templates,`dynamic_templates`>>.
 
 [[date-detection]]
 ==== Date detection
@@ -129,4 +131,3 @@ PUT my-index-000001/_doc/1
 
 <1> The `my_float` field is added as a <<number,`float`>> field.
 <2> The `my_integer` field is added as a <<number,`long`>> field.
-