Introduce hidden indices (elastic#50452)

This change introduces a new feature for indices so that they can be
hidden from wildcard expansion. The feature is referred to as hidden
indices. An index can be marked hidden through the use of an index
setting, `index.hidden`, at creation time. One primary use case for
this feature is to have a construct that fits indices that are created
by the stack that contain data used for display to the user and/or
intended for querying by the user. The desire to keep them hidden is
to avoid confusing users when searching all of the data they have
indexed and getting results returned from indices created by the
system.

Hidden indices have the following properties:
* API calls for all indices (empty indices array, _all, or *) will not
  return hidden indices by default.
* Wildcard expansion will not return hidden indices by default unless
  the wildcard pattern begins with a `.`. This behavior is similar to
  shell expansion of wildcards.
* REST API calls can enable the expansion of wildcards to hidden
  indices with the `expand_wildcards` parameter. To expand wildcards
  to hidden indices, use the value `hidden` in conjunction with `open`
  and/or `closed`.
* Creation of a hidden index will ignore global index templates. A
  global index template is one with a match-all pattern.
* Index templates can make an index hidden, with the exception of a
  global index template.
* Accessing a hidden index directly requires no additional parameters.

Relates elastic#50251

Author: jaymode

docs/reference/index-modules.asciidoc

@@ -79,6 +79,13 @@ indices.
     Indicates whether <<query-filter-context, cached filters>> are pre-loaded for
     nested queries. Possible values are `true` (default) and `false`.
 
+`index.hidden`::
+
+    Indicates whether the index should be hidden by default. Hidden indices are not
+    returned by default when using a wildcard expression. This behavior is controlled
+    per request through the use of the `expand_wildcards` parameter. Possible values are
+    `true` and `false` (default).
+
 [float]
 [[dynamic-index-settings]]
 === Dynamic index settings

docs/reference/rest-api/common-parms.asciidoc

@@ -53,7 +53,7 @@ Specifies what to do when the request:
 --
 * Contains wildcard expressions and there are no {transforms} that match.
 * Contains the `_all` string or no identifiers and there are no matches.
-* Contains wildcard expressions and there are only partial matches. 
+* Contains wildcard expressions and there are only partial matches.
 
 The default value is `true`, which returns an empty `transforms` array when
 there are no matches and the subset of results when there are partial matches.
@@ -69,7 +69,7 @@ Specifies what to do when the request:
 --
 * Contains wildcard expressions and there are no {transforms} that match.
 * Contains the `_all` string or no identifiers and there are no matches.
-* Contains wildcard expressions and there are only partial matches. 
+* Contains wildcard expressions and there are only partial matches.
 
 The default value is `true`, which returns a successful acknowledgement message
 when there are no matches. When there are only partial matches, the API stops
@@ -90,7 +90,7 @@ end::analyzer[]
 
 tag::analyze_wildcard[]
 `analyze_wildcard`::
-(Optional, boolean) If `true`, wildcard and prefix queries are 
+(Optional, boolean) If `true`, wildcard and prefix queries are
 analyzed. Defaults to `false`.
 end::analyze_wildcard[]
 
@@ -118,7 +118,7 @@ end::completion-fields[]
 
 tag::default_operator[]
 `default_operator`::
-(Optional, string) The default operator for query string query: AND or OR. 
+(Optional, string) The default operator for query string query: AND or OR.
 Defaults to `OR`.
 end::default_operator[]
 
@@ -144,7 +144,7 @@ end::detailed[]
 
 tag::df[]
 `df`::
-(Optional, string) Field to use as default where no field prefix is 
+(Optional, string) Field to use as default where no field prefix is
 given in the query string.
 end::df[]
 
@@ -181,14 +181,17 @@ Expand only to open indices.
 `closed`::
 Expand only to closed indices.
 
+`hidden`::
+Expansion of wildcards will include hidden indices.
+
 `none`::
 Wildcard expressions are not accepted.
 --
 end::expand-wildcards[]
 
 tag::field_statistics[]
 `field_statistics`::
-(Optional, boolean) If `true`, the response includes the document count, sum of document frequencies, 
+(Optional, boolean) If `true`, the response includes the document count, sum of document frequencies,
 and sum of total term frequencies.
 Defaults to `true`.
 end::field_statistics[]
@@ -297,9 +300,9 @@ end::help[]
 
 tag::bulk-id[]
 `_id`::
-(Optional, string) 
+(Optional, string)
 The document ID.
-If no ID is specified, a document ID is automatically generated. 
+If no ID is specified, a document ID is automatically generated.
 end::bulk-id[]
 
 tag::if_primary_term[]
@@ -316,7 +319,7 @@ end::if_seq_no[]
 
 tag::ignore_throttled[]
 `ignore_throttled`::
-(Optional, boolean) If `true`, concrete, expanded or aliased indices are 
+(Optional, boolean) If `true`, concrete, expanded or aliased indices are
 ignored when throttled.
 end::ignore_throttled[]
 
@@ -362,7 +365,7 @@ end::index[]
 tag::bulk-index[]
 `_index`::
 (Optional, string)
-The name of the target index. 
+The name of the target index.
 Required if not specified as a path parameter.
 end::bulk-index[]
 
@@ -447,7 +450,7 @@ end::index-template[]
 
 tag::lenient[]
 `lenient`::
-(Optional, boolean) If `true`, format-based query failures (such as 
+(Optional, boolean) If `true`, format-based query failures (such as
 providing text to a numeric field) will be ignored. Defaults to `false`.
 end::lenient[]
 
@@ -618,13 +621,13 @@ end::search-q[]
 
 tag::query[]
 `query`::
-(Optional, <<query-dsl,query object>>) Defines the search definition using the 
+(Optional, <<query-dsl,query object>>) Defines the search definition using the
 <<query-dsl,Query DSL>>.
 end::query[]
 
 tag::realtime[]
 `realtime`::
-(Optional, boolean) If `true`, the request is real-time as opposed to near-real-time. 
+(Optional, boolean) If `true`, the request is real-time as opposed to near-real-time.
 Defaults to `true`. See <<realtime>>.
 end::realtime[]
 
@@ -638,7 +641,7 @@ end::refresh[]
 
 tag::request_cache[]
 `request_cache`::
-(Optional, boolean) If `true`, the request cache is used for this request. 
+(Optional, boolean) If `true`, the request cache is used for this request.
 Defaults to the index-level setting.
 end::request_cache[]
 
@@ -668,14 +671,14 @@ end::cat-s[]
 
 tag::scroll[]
 `scroll`::
-(Optional, <<time-units, time units>>) Specifies how long a consistent view of 
+(Optional, <<time-units, time units>>) Specifies how long a consistent view of
 the index should be maintained for scrolled search.
 end::scroll[]
 
 tag::scroll_size[]
 `scroll_size`::
-(Optional, integer) Size of the scroll request that powers the operation. 
-Defaults to 100. 
+(Optional, integer) Size of the scroll request that powers the operation.
+Defaults to 100.
 end::scroll_size[]
 
 tag::search_timeout[]
@@ -731,7 +734,7 @@ end::size-transforms[]
 
 tag::slices[]
 `slices`::
-(Optional, integer) The number of slices this task should be divided into. 
+(Optional, integer) The number of slices this task should be divided into.
 Defaults to 1 meaning the task isn't sliced into subtasks.
 end::slices[]
 
@@ -742,24 +745,24 @@ end::sort[]
 
 tag::source[]
 `_source`::
-(Optional, string) True or false to return the `_source` field or not, or a 
+(Optional, string) True or false to return the `_source` field or not, or a
 list of fields to return.
 end::source[]
 
 tag::source_excludes[]
 `_source_excludes`::
-(Optional, string) A list of fields to exclude from the returned `_source` 
+(Optional, string) A list of fields to exclude from the returned `_source`
 field.
 end::source_excludes[]
 
 tag::source_includes[]
 `_source_includes`::
-(Optional, string) A list of fields to extract and return from the `_source` 
+(Optional, string) A list of fields to extract and return from the `_source`
 field.
 end::source_includes[]
 
 tag::source-transforms[]
-The source of the data for the {transform}. 
+The source of the data for the {transform}.
 end::source-transforms[]
 
 tag::source-index-transforms[]
@@ -823,13 +826,13 @@ end::task-id[]
 
 tag::term_statistics[]
 `term_statistics`::
-(Optional, boolean) If `true`, the response includes term frequency and document frequency. 
+(Optional, boolean) If `true`, the response includes term frequency and document frequency.
 Defaults to `false`.
 end::term_statistics[]
 
 tag::terminate_after[]
 `terminate_after`::
-(Optional, integer) The maximum number of documents to collect for each shard, 
+(Optional, integer) The maximum number of documents to collect for each shard,
 upon reaching which the query execution will terminate early.
 end::terminate_after[]
 
@@ -871,7 +874,7 @@ end::transform-id-wildcard[]
 
 tag::cat-v[]
 `v`::
-(Optional, boolean) If `true`, the response includes column headings. 
+(Optional, boolean) If `true`, the response includes column headings.
 Defaults to `false`.
 end::cat-v[]
 
@@ -912,6 +915,6 @@ end::wait_for_active_shards[]
 
 tag::wait_for_completion[]
 `wait_for_completion`::
-(Optional, boolean) If `true`, the request blocks until the operation is complete. 
+(Optional, boolean) If `true`, the request blocks until the operation is complete.
 Defaults to `true`.
 end::wait_for_completion[]

server/src/main/java/org/elasticsearch/action/admin/cluster/health/ClusterHealthRequest.java

@@ -39,7 +39,7 @@
 public class ClusterHealthRequest extends MasterNodeReadRequest<ClusterHealthRequest> implements IndicesRequest.Replaceable {
 
     private String[] indices;
-    private IndicesOptions indicesOptions = IndicesOptions.lenientExpand();
+    private IndicesOptions indicesOptions = IndicesOptions.lenientExpandHidden();
     private TimeValue timeout = new TimeValue(30, TimeUnit.SECONDS);
     private ClusterHealthStatus waitForStatus;
     private boolean waitForNoRelocatingShards = false;

server/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java

@@ -43,7 +43,6 @@
 import org.elasticsearch.cluster.metadata.MetaData;
 import org.elasticsearch.cluster.metadata.MetaDataCreateIndexService;
 import org.elasticsearch.cluster.metadata.MetaDataIndexAliasesService;
-import org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService;
 import org.elasticsearch.cluster.service.ClusterService;
 import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.inject.Inject;
@@ -63,6 +62,8 @@
 import java.util.regex.Pattern;
 import java.util.stream.Collectors;
 
+import static org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.findTemplates;
+
 /**
  * Main class to swap the index pointed to by an alias, given some conditions
  */
@@ -122,7 +123,9 @@ protected void masterOperation(Task task, final RolloverRequest rolloverRequest,
             : generateRolloverIndexName(sourceProvidedName, indexNameExpressionResolver);
         final String rolloverIndexName = indexNameExpressionResolver.resolveDateMathExpression(unresolvedName);
         MetaDataCreateIndexService.validateIndexName(rolloverIndexName, state); // will fail if the index already exists
-        checkNoDuplicatedAliasInIndexTemplate(metaData, rolloverIndexName, rolloverRequest.getAlias());
+        final Boolean isHidden = IndexMetaData.INDEX_HIDDEN_SETTING.exists(rolloverRequest.getCreateIndexRequest().settings()) ?
+            IndexMetaData.INDEX_HIDDEN_SETTING.get(rolloverRequest.getCreateIndexRequest().settings()) : null;
+        checkNoDuplicatedAliasInIndexTemplate(metaData, rolloverIndexName, rolloverRequest.getAlias(), isHidden);
         IndicesStatsRequest statsRequest = new IndicesStatsRequest().indices(rolloverRequest.getAlias())
             .clear()
             .indicesOptions(IndicesOptions.fromOptions(true, false, true, true))
@@ -291,8 +294,9 @@ static CreateIndexClusterStateUpdateRequest prepareCreateIndexRequest(final Stri
      * the rollover alias will point to multiple indices. This causes indexing requests to be rejected.
      * To avoid this, we make sure that there is no duplicated alias in index templates before creating a new index.
      */
-    static void checkNoDuplicatedAliasInIndexTemplate(MetaData metaData, String rolloverIndexName, String rolloverRequestAlias) {
-        final List<IndexTemplateMetaData> matchedTemplates = MetaDataIndexTemplateService.findTemplates(metaData, rolloverIndexName);
+    static void checkNoDuplicatedAliasInIndexTemplate(MetaData metaData, String rolloverIndexName, String rolloverRequestAlias,
+                                                      @Nullable Boolean isHidden) {
+        final List<IndexTemplateMetaData> matchedTemplates = findTemplates(metaData, rolloverIndexName, isHidden);
         for (IndexTemplateMetaData template : matchedTemplates) {
             if (template.aliases().containsKey(rolloverRequestAlias)) {
                 throw new IllegalArgumentException(String.format(Locale.ROOT,

server/src/main/java/org/elasticsearch/action/bulk/TransportBulkAction.java

@@ -307,7 +307,7 @@ static boolean resolvePipelines(final DocWriteRequest<?> originalRequest, final
                 }
             } else if (indexRequest.index() != null) {
                 // the index does not exist yet (and this is a valid request), so match index templates to look for pipelines
-                List<IndexTemplateMetaData> templates = MetaDataIndexTemplateService.findTemplates(metaData, indexRequest.index());
+                List<IndexTemplateMetaData> templates = MetaDataIndexTemplateService.findTemplates(metaData, indexRequest.index(), null);
                 assert (templates != null);
                 // order of templates are highest order first
                 for (final IndexTemplateMetaData template : templates) {

server/src/main/java/org/elasticsearch/action/search/MultiSearchRequest.java

@@ -23,6 +23,7 @@
 import org.elasticsearch.action.ActionRequestValidationException;
 import org.elasticsearch.action.CompositeIndicesRequest;
 import org.elasticsearch.action.support.IndicesOptions;
+import org.elasticsearch.action.support.IndicesOptions.WildcardStates;
 import org.elasticsearch.common.CheckedBiConsumer;
 import org.elasticsearch.common.bytes.BytesReference;
 import org.elasticsearch.common.io.stream.StreamInput;
@@ -304,15 +305,7 @@ public static void writeSearchRequestParams(SearchRequest request, XContentBuild
             xContentBuilder.field("index", request.indices());
         }
         if (request.indicesOptions() != null && request.indicesOptions() != SearchRequest.DEFAULT_INDICES_OPTIONS) {
-            if (request.indicesOptions().expandWildcardsOpen() && request.indicesOptions().expandWildcardsClosed()) {
-                xContentBuilder.field("expand_wildcards", "all");
-            } else if (request.indicesOptions().expandWildcardsOpen()) {
-                xContentBuilder.field("expand_wildcards", "open");
-            } else if (request.indicesOptions().expandWildcardsClosed()) {
-                xContentBuilder.field("expand_wildcards", "closed");
-            } else {
-                xContentBuilder.field("expand_wildcards", "none");
-            }
+            WildcardStates.toXContent(request.indicesOptions().getExpandWildcards(), xContentBuilder);
             xContentBuilder.field("ignore_unavailable", request.indicesOptions().ignoreUnavailable());
             xContentBuilder.field("allow_no_indices", request.indicesOptions().allowNoIndices());
         }