[DOCS] Document hidden data streams and naming restrictions

[jrodewig]

Documents #63987 and #65795

Previews

• API conventions: https://elasticsearch_65862.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/multi-index.html#hidden-indices
• Set up a data stream: https://elasticsearch_65862.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/set-up-a-data-stream.html#create-a-data-stream-template
• Create a data stream API: https://elasticsearch_65862.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/indices-create-data-stream.html#indices-create-data-stream-api-path-params
• Data stream stats API: https://elasticsearch_65862.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/data-stream-stats-api.html#data-stream-stats-api-query-params
• Delete data stream API: https://elasticsearch_65862.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/indices-delete-data-stream.html#delete-data-stream-api-query-params
• Get data stream API: https://elasticsearch_65862.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/indices-get-data-stream.html#get-data-stream-api-query-parms
• Put index template API: https://elasticsearch_65862.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/indices-put-template.html#put-index-template-api-request-body

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[jrodewig]

Sorry for the false start. This is ready for review at your convenience @probakowski.

[jrodewig]

@elasticmacine update branch

[jrodewig]

Hi @probakowski! Can you let me know what you think of these changes? I'd love to get these docs in before the 7.11 release. Thanks!

[danhermann]

In my testing, open and closed match hidden indices and data streams though I'm not sure why.

[jrodewig]

Thanks @danhermann. That sounds like a big regression, but I wasn't able to reproduce. Here's how I tested when running the latest master branch locally using ./gradlew run -Dtests.es.xpack.security.enabled=false:

### 1. Create a non-hidden data stream.

PUT _index_template/visible-data-stream-template
{
  "index_patterns": [ "visible-data-stream" ],
  "data_stream": { },
  "priority": 200
}
PUT _data_stream/visible-data-stream

### 2. Create hidden data stream.

PUT _index_template/hidden-data-stream-template
{
  "index_patterns": [ "hidden-data-stream" ],
  "data_stream": {
    "hidden": true
  },
  "priority": 200
}
PUT _data_stream/hidden-data-stream

### 3. Get request with `expand_wildcards=open`.

GET *-data-stream?expand_wildcards=open

# Returns info for `visible-data-stream` only.

### 4. Get data streams with `expand_wildcards=open,hidden`.

GET *-data-stream?expand_wildcards=open,hidden

# Returns info for `visible-data-stream` and `hidden-data-stream`.

### 5. Create an open, hidden index.

PUT hidden-index
{
  "settings": {
    "index.hidden": true
  }
}

### 6. Create and close an index.

PUT closed-index
POST closed-index/_close

### 7. Get request with `expand_wildcards=closed`.

GET *-data-stream,*-index?expand_wildcards=closed

# Returns info for `closed-index` only.

[danhermann]

Hmm, interesting. I was testing primarily with * as the wildcard. With a more restrictive wildcard, it does appear to work as you've described.

[probakowski]

Thanks @jrodewig and sorry for the very long delay here. Overall changes look great, left couple comments about open/closed for data streams

[probakowski]

For data streams API (get, delete, stats etc) both open and closed will return all data streams (excluding hidden with hidden is not specified) - we don't have a concept of closed and open data streams. I'm not really sure how to point that out in the documentation

[jrodewig]

Good point. I was initially side-stepping this, but it may be better to be explicit.

With 04cd440, we now note that data streams cannot be closed. I don't think we need that for open, but let me know.

[probakowski]

For data streams API (get, delete, stats etc) there's a shortcut - if you provide only hidden it will work as open,hidden - we don't have a concept of closed and open data streams so forcing user to add it seems unnecessary. Of course it's not the case for search APIs where open and closed apply to the backing indices of data stream.

[jrodewig]

To address this in 04cd440, I added a new definition specifically for the data stream APIs.

[jrodewig]

Thanks for the review @probakowski. I made some changes to address your comments. Please let me know if you have any other feedback.

[probakowski]

LGTM, thanks @jrodewig !