Skip to content

[DOCS] Document hidden data streams and naming restrictions #65862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jan 13, 2021
Merged

[DOCS] Document hidden data streams and naming restrictions #65862

merged 12 commits into from
Jan 13, 2021

Conversation

jrodewig
Copy link
Contributor

@jrodewig jrodewig commented Dec 3, 2020

@jrodewig jrodewig requested a review from probakowski December 4, 2020 00:40
@jrodewig jrodewig added :Data Management/Data streams Data streams and their lifecycles >docs General docs changes labels Dec 4, 2020
@jrodewig jrodewig marked this pull request as ready for review December 4, 2020 00:40
@elasticmachine elasticmachine added Team:Data Management Meta label for data/management team Team:Docs Meta label for docs team labels Dec 4, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (Team:Core/Features)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-docs (Team:Docs)

@jrodewig jrodewig marked this pull request as draft December 4, 2020 01:34
@jrodewig jrodewig marked this pull request as ready for review December 4, 2020 02:19
@jrodewig
Copy link
Contributor Author

jrodewig commented Dec 4, 2020

Sorry for the false start. This is ready for review at your convenience @probakowski.

@jrodewig
Copy link
Contributor Author

jrodewig commented Dec 8, 2020

@elasticmacine update branch

@jrodewig
Copy link
Contributor Author

jrodewig commented Jan 6, 2021

Hi @probakowski! Can you let me know what you think of these changes? I'd love to get these docs in before the 7.11 release. Thanks!


`open`::
Expand only to open indices.
Match non-hidden data streams and open, non-hidden indices.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my testing, open and closed match hidden indices and data streams though I'm not sure why.

Copy link
Contributor Author

@jrodewig jrodewig Jan 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @danhermann. That sounds like a big regression, but I wasn't able to reproduce. Here's how I tested when running the latest master branch locally using ./gradlew run -Dtests.es.xpack.security.enabled=false:

### 1. Create a non-hidden data stream.

PUT _index_template/visible-data-stream-template
{
  "index_patterns": [ "visible-data-stream" ],
  "data_stream": { },
  "priority": 200
}
PUT _data_stream/visible-data-stream

### 2. Create hidden data stream.

PUT _index_template/hidden-data-stream-template
{
  "index_patterns": [ "hidden-data-stream" ],
  "data_stream": {
    "hidden": true
  },
  "priority": 200
}
PUT _data_stream/hidden-data-stream

### 3. Get request with `expand_wildcards=open`.

GET *-data-stream?expand_wildcards=open

# Returns info for `visible-data-stream` only.

### 4. Get data streams with `expand_wildcards=open,hidden`.

GET *-data-stream?expand_wildcards=open,hidden

# Returns info for `visible-data-stream` and `hidden-data-stream`.

### 5. Create an open, hidden index.

PUT hidden-index
{
  "settings": {
    "index.hidden": true
  }
}

### 6. Create and close an index.

PUT closed-index
POST closed-index/_close

### 7. Get request with `expand_wildcards=closed`.

GET *-data-stream,*-index?expand_wildcards=closed

# Returns info for `closed-index` only.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, interesting. I was testing primarily with * as the wildcard. With a more restrictive wildcard, it does appear to work as you've described.

Copy link
Contributor

@probakowski probakowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jrodewig and sorry for the very long delay here. Overall changes look great, left couple comments about open/closed for data streams


`closed`::
Expand only to closed indices.
Match closed indices.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For data streams API (get, delete, stats etc) both open and closed will return all data streams (excluding hidden with hidden is not specified) - we don't have a concept of closed and open data streams. I'm not really sure how to point that out in the documentation

Copy link
Contributor Author

@jrodewig jrodewig Jan 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I was initially side-stepping this, but it may be better to be explicit.

With 04cd440, we now note that data streams cannot be closed. I don't think we need that for open, but let me know.

Expansion of wildcards will include hidden indices.
Must be combined with `open`, `closed`, or both.
Match hidden data streams and hidden indices. Must be combined with `open`,
`closed`, or both.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For data streams API (get, delete, stats etc) there's a shortcut - if you provide only hidden it will work as open,hidden - we don't have a concept of closed and open data streams so forcing user to add it seems unnecessary. Of course it's not the case for search APIs where open and closed apply to the backing indices of data stream.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To address this in 04cd440, I added a new definition specifically for the data stream APIs.

@jrodewig
Copy link
Contributor Author

Thanks for the review @probakowski. I made some changes to address your comments. Please let me know if you have any other feedback.

@jrodewig jrodewig requested a review from probakowski January 12, 2021 15:19
Copy link
Contributor

@probakowski probakowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @jrodewig !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Data streams Data streams and their lifecycles >docs General docs changes Team:Data Management Meta label for data/management team Team:Docs Meta label for docs team v7.11.0 v7.12.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants