[DOCS] Document hidden data streams and naming restrictions (#65862) (#67434)

Author: jrodewig

docs/reference/api-conventions.asciidoc

@@ -52,14 +52,19 @@ NOTE: Single index APIs, such as the <<docs>> and
 syntax.
 
 [[hidden-indices]]
-==== Hidden indices
+==== Hidden data streams and indices
 
-Indices that are configured to be hidden with the <<index-hidden,index.hidden>> setting are 
-excluded from mult-target queries by default. 
-To include hidden indices, you must specify the `expand_wildcards` parameter.
+For most APIs, wildcard expressions do not match hidden data streams and indices
+by default. To match hidden data streams and indices using a wildcard
+expression, you must specify the `expand_wildcards` query parameter.
 
-The backing indices for data streams are hidden indices, 
-and some features like {ml} store information in hidden indices. 
+You can create hidden data streams by setting
+<<data-stream-hidden,`data_stream.hidden`>> to `true` in the stream's matching
+index template. You can hide indices using the <<index-hidden,`index.hidden`>>
+index setting.
+
+The backing indices for data streams are hidden automatically. Some features,
+such as {ml}, store information in hidden indices.
 
 Global index templates that match all indices are not applied to hidden indices.
 

docs/reference/data-streams/set-up-a-data-stream.asciidoc

@@ -66,7 +66,8 @@ PUT /_ilm/policy/my-data-stream-policy
 template is used for data streams.
 . Use the wizard to finish defining your template. Specify:
 
-* One or more index patterns that match the data stream's name.
+* One or more index patterns that match the data stream's name. +
+include::{es-repo-dir}/indices/create-data-stream.asciidoc[tag=data-stream-name]
 
 * Mappings and settings for the stream's backing indices.
 
@@ -94,7 +95,6 @@ maps `@timestamp` as a `date` field  with default options.
 
 If using {ilm-init}, specify your lifecycle policy in the `index.lifecycle.name`
 setting.
-
 TIP: Carefully consider your template's mappings and settings. Later changes may
 require reindexing. See <<data-streams-change-mappings-and-settings>>.
 
@@ -105,7 +105,7 @@ image::images/data-streams/create-index-template.png[Create template page]
 .API example
 ====
 Use the <<indices-put-template,put index template API>> to create an index
-template. The template must include an empty `data_stream` object, indicating
+template. The template must include a `data_stream` object, indicating
 it's used for data streams.
 
 [source,console]

docs/reference/indices/create-data-stream.asciidoc

@@ -49,14 +49,16 @@ DELETE /_index_template/template
 --
 (Required, string) Name of the data stream to create.
 
+// tag::data-stream-name[]
 Data stream names must meet the following criteria:
 
 - Lowercase only
-- Cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, ` ` (space character),
-`,`, `#`, `:`
-- Cannot start with `-`, `_`, `+`, `.`
+- Cannot include `\`, `/`, `*`, `?`, `"`, `<`, `>`, `|`, `,`, `#`, `:`, or a
+space character
+- Cannot start with `-`, `_`, `+`, or `.ds-`
 - Cannot be `.` or `..`
-- Cannot be longer than 255 bytes (note it is bytes, so multi-byte characters
-will count towards the 255 limit faster)
+- Cannot be longer than 255 bytes. Multi-byte characters
+count towards this limit faster.
+// end::data-stream-name[]
 --
 

docs/reference/indices/data-stream-stats.asciidoc

@@ -72,6 +72,10 @@ To target all data streams in a cluster, omit this parameter or use `*`.
 [[data-stream-stats-api-query-params]]
 ==== {api-query-parms-title}
 
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=ds-expand-wildcards]
++
+Defaults to `open`.
+
 `human`::
 (Optional, Boolean)
 If `true`, the response includes statistics in human-readable <<byte-units,byte

docs/reference/indices/delete-data-stream.asciidoc

@@ -48,3 +48,11 @@ DELETE /_index_template/template
 (Required, string)
 Comma-separated list of data streams to delete.
 Wildcard (`*`) expressions are supported.
+
+[role="child_attributes"]
+[[delete-data-stream-api-query-parms]]
+==== {api-query-parms-title}
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=ds-expand-wildcards]
++
+Defaults to `open`.

docs/reference/indices/get-data-stream.asciidoc

@@ -84,6 +84,14 @@ Comma-separated list of data stream names used to limit the request. Wildcard
 (`*`) expressions are supported. If omitted, all data streams will be
 returned.
 
+[role="child_attributes"]
+[[get-data-stream-api-query-parms]]
+==== {api-query-parms-title}
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=ds-expand-wildcards]
++
+Defaults to `open`.
+
 [role="child_attributes"]
 [[get-data-stream-api-response-body]]
 ==== {api-response-body-title}
@@ -184,6 +192,10 @@ included in the response.
 NOTE: A data stream's backing indices may be assigned different lifecycle
 policies. To retrieve the lifecycle policy for individual backing indices,
 use the <<indices-get-settings,get index settings API>>.
+
+`hidden`::
+(Boolean)
+If `true`, the data stream is <<hidden-indices,hidden>>.
 ====
 
 [[get-data-stream-api-example]]

docs/reference/indices/put-index-template.asciidoc

@@ -103,12 +103,20 @@ your template is applied instead of the built-in template for `logs-*-*`.
 
 [xpack]#`data_stream`#::
 (Optional, object)
-Indicates whether the template is used to create data streams and their backing
-indices. If so, use an empty object as the argument: +
-`data_stream: { }`.
+If this object is included, the template is used to create data streams and
+their backing indices. Supports an empty object: `data_stream: { }`
 +
 Data streams require a matching index template with a `data_stream` object.
 See <<create-a-data-stream-template>>.
++
+.Properties of `data_stream`
+[%collapsible%open]
+====
+[[data-stream-hidden]]
+`hidden`::
+(Optional, Boolean)
+If `true`, the data stream is <<hidden-indices,hidden>>. Defaults to `false`.
+====
 
 `template`::
 (Optional, object)

docs/reference/rest-api/common-parms.asciidoc

@@ -215,26 +215,48 @@ Enrich policy name
 used to limit the request.
 end::enrich-policy[]
 
+tag::ds-expand-wildcards[]
+`expand_wildcards`::
++
+--
+(Optional, string)
+Type of data stream that wildcard expressions can match. Supports
+comma-separated values, such as `open,hidden`. Valid values are:
+
+`all`, `hidden`::
+Match any data stream, including <<hidden-indices,hidden>> ones.
+
+`open`, `closed`::
+Matches any non-hidden data stream. Data streams cannot be closed.
+
+`none`::
+Wildcard expressions are not accepted.
+--
+end::ds-expand-wildcards[]
+
 tag::expand-wildcards[]
 `expand_wildcards`::
 +
 --
-(Optional, string) Controls what kind of indices that wildcard expressions can
-expand to. Multiple values are accepted when separated by a comma, as in
-`open,hidden`. Valid values are:
+(Optional, string)
+Type of index that wildcard expressions can match. If the request can target
+data streams, this argument determines whether wildcard expressions match
+hidden data streams. Supports comma-separated values, such as `open,hidden`.
+Valid values are:
 
 `all`::
-Expand to open and closed indices, including <<hidden-indices,hidden indices>>.
+Match any data stream or index, including <<hidden-indices,hidden>> ones.
 
 `open`::
-Expand only to open indices.
+Match open, non-hidden indices. Also matches any non-hidden data stream.
 
 `closed`::
-Expand only to closed indices.
+Match closed, non-hidden indices. Also matches any non-hidden data stream. Data
+streams cannot be closed.
 
 `hidden`::
-Expansion of wildcards will include hidden indices.
-Must be combined with `open`, `closed`, or both.
+Match hidden data streams and hidden indices. Must be combined with `open`,
+`closed`, or both.
 
 `none`::
 Wildcard expressions are not accepted.