Review system indices documentation for simplicity and consistency

[williamrandolph]

A goal of the system indices project is to, as much as possible, allow users not to care about system indices (and system data streams). We are not there yet, so we need to figure out how to explain what system indices are and provide enough information to troubleshoot issues that users might have with them, without getting into implementation details.

We have an item about system indices on our 8.x API Conventions page. It reads:

Elasticsearch modules and plugins can store configuration and state information in internal system indices. You should not directly access or modify system indices as they contain data essential to the operation of the system.

Here are some situations where a user might interact with system indices:

1. The negative case: "I just upgraded, but where did my .kibana and .security indices go? Did I lose them in the upgrade?" The answer: no, they just became hidden system indices and you can't see them anymore.
2. The investigation case: "I'm seeing log messages related to an index, but it's not one that I created and not one that I can view in the API." In order to find out what's going on, you will need superuser privileges.
3. The privilege hack: "We keep having trouble with this one system index, so I'd like to give my non-superuser admins permission to view and investigate it." In this case, you would want to create a new security role with an index permission that targets the index in question and has allow_restricted_indices set to true. (See Defining Roles: Indices Privileges.)
4. The indirect exposure case: Several diagnostic or informational APIs do nothing to hide the existence of system indices. For example, the _cat/shards API will show shards for system indices as well as for normal indices, and system indices are listed in snapshot data.

Finally, we may want to clarify that for the superuser, system indices are like hidden indices, but they have certain additional protection around their settings and mappings.

We may also want to warn about the case where legacy templates that target * will be applied to system indices, which can sometimes break a cluster in ways that are hard to diagnose.

I'm not exactly sure how to put all this together in the documents, but, basically, we need to give users enough information that, when they run into system indices somehow, they can figure out what's going on.

cc @gwbrown, @debadair, @lockewritesdocs

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[damienalexandre]

Jumping on the discussion (sorry if off-topic) but have you considered at some point using the _meta index mapping field to add a description of system indices?

I'm in a situation where I feel spammed by those indices and it looks to me a lot of them have no purpose in my cluster; getting a short description for each could help, to know if they are safe to delete, or what they are used for.

[jrodewig]

As @gwbrown pointed out, we should probably formalize definitions for related concepts like dot indices and regular indices. I created #81531 to track that work.

[Leaf-Lin]

Is there a setting in the index itself to say it's a system one or not? Because AFAIK, all system indices are dot indices, but not all dot indices are system indices.

Today, we already have a setting hidden in the index to indicate whether it's hidden by default. Is it possible to have the following to verify whether it's a system or a regular dot?

   "settings" : {
        "hidden" : "true",
        "system" : "true"   <--- proposed addition

[gwbrown]

There isn't a setting and adding one might be a bit of a headache, but we do have a boolean in the index metadata. It's not currently visible in the Get Index or _cat/indices APIs, but until we can add it to those APIs, you can at least see it using GET _cluster/state/metadata/<index_name> (or _cluster/state, but it's much easier to read if you get ES to do the filtering for you). For example, for .tasks:

{
    "cluster_name": "runTask",
    "cluster_uuid": "phWu8kXERXCu567gNQjjww",
    "metadata": {
        "cluster_uuid": "phWu8kXERXCu567gNQjjww",
        "cluster_uuid_committed": false,
        "cluster_coordination": {
            "term": 1,
            "last_committed_config": [
                "8IWt0NXFRvK1aB6i0AorKQ"
            ],
            "last_accepted_config": [
                "8IWt0NXFRvK1aB6i0AorKQ"
            ],
            "voting_config_exclusions": []
        },
        "templates": {},
        "indices": {
            ".tasks": {
                "version": 6,
                "mapping_version": 1,
                "settings_version": 2,
                "aliases_version": 1,
                "routing_num_shards": 1024,
                "state": "open",
                "settings": {"elided_for_brevity": true},
                "mappings": {"elided_for_brevity": true},
                "aliases": [],
                "primary_terms": {
                    "0": 1
                },
                "in_sync_allocations": {
                    "0": [
                        "tmB_mz1cTFWsmSh2Edv2VQ"
                    ]
                },
                "rollover_info": {},
                "system": true, // <---- Here's the system flag
                "timestamp_range": {
                    "unknown": true
                }
            }
        },
        "index-graveyard": {
            "tombstones": []
        }
    }
}

[lucabelluccini]

Thank you @gwbrown for the tip. A short request to get if indices are system or not:

GET _cluster/state/metadata?filter_path=metadata.indices.*.system