Add a min_docs parameter for rollover action in ILM

[raf64flo]

Hi,

This feature request is related to this forum thread: https://discuss.elastic.co/t/what-about-a-min-docs-parameter-for-rollover-action-in-index-lifecycle-management/195242

This issue precises further the request and the use cases discussed in the forum.

The rollover action currently allows for triggering a new index using 3 parameters:

Name	Required	Default	Description
max_size	no	-	max primary shard index storage size. See Byte Units for formatting
max_docs	no	-	max number of documents an index is to contain before rolling over.
max_age	no	-	max time elapsed from index creation. See Time Units for formatting

Use cases

The Rollover action currently is really useful in time series indices, but can also be used for indices (named below datastore indices) which are not time series (by avoiding the max_age condition).

For both contexts there are cases for which we can end with lots of empty or very small indices when using the max_age condition, if the indexing throughput is low or null during some period of time. This can lead to a very big number of shards in the cluster and makes potentially its stability at risk.

The delete action could be used to balance this risk, but there are contexts where the index deletion is not related to the index lifecycle directly, and could be related to an external condition (unrelated to the index age nor its size). Hence, we should be able to use safely the rollover action without the need to implement a delete action.

Time series indices

In the case of time series indices, we can have some period of very low throughput, or even no throughput at all, so using max_age makes a new index created each hour (in the example below), even if no documents have been added in the given period, leading to a series of useless very small or empty indices.

Datastore indices

During an heavy indexing load, we are able to use the ILM policy, in order to keep index size in safe limits, automatically. We can do this by using the max_docs or max_size condition to trigger the rollover. But for the last X documents (X being lower than max_docs and having size lower than max_size), they will stay indefinitely in the hot phase, even if we know the indexing process is finished, so they cannot benefit of the next cleaning processes (such as shrink, forcemerge and so on) in the warm phase of the defined policy.

Then, if we use the max_age condition to prevent the last indexed documents from staying in the hot index, the number of empty indices will increase over time, unnecessary, after the indexing process is finished.

Feature request

To prevent the creation of any empty (or too small) index, we could add a min_docs condition to be used in conjunction with the other conditions, as following:

Name	Required	Default	Description
min_docs	no	0	min number of documents an index is to contain before rolling over. This condition is considered in conjunction with any other condition.

Example:

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "conditions": {
              "max_age":   "1h",
              "min_docs":   "1000",
              "max_docs":  100000,
              "max_size":  "5gb"
            }
          }
        }
      }
    }
  }
}

Constraints on the new condition

• 0 <= min_docs < max_docs
• default value = 0, in order to keep compatibility with previous behavior
• contrary to the other conditions which trigger the action as soon as they are met (according to the indices.lifecycle.poll_interval), this one should be considered in conjunction with the others and could not be used alone.
• as a consequence, using min_docs as a unique condition should be invalid and throw an exception.

API definition

To reflect this requirement, we could change the way the conditions are declared in several manners, in addition to the example above:

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "conditions": {
              "or": {
                "max_age": "1h",
                "max_docs": 100000,
                "max_size": "5gb"
              },
              "and": {
                "min_docs": "1000"
              }
            }
          }
        }
      }
    }
  }
}

This way is clearer in my mind to explicit how the triggers are met, but it has the cons to change the API.

To keep compatibility, we could move to something like:

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "conditions": {
              "max_age": "1h",
              "max_docs": 100000,
              "max_size": "5gb",
              "and_min_docs": "1000"
            }
          }
        }
      }
    }
  }
}

Using an explicit and_min_docs condition that could not be met alone, without at least one other condition.

[elasticmachine]

Pinging @elastic/es-core-features

[gwbrown]

I've changed the tag on this from ILM to Indices APIs, as most of this change would be to the Rollover API - the only ILM changes needed would be to make sure the parameter is passed through to the Rollover API.

[gwbrown]

We discussed this in the Core/Features meeting today - here's the results of that discussion:
This seems to be a relatively rare use case. Adding this feature would add significant complexity to the Rollover API, which has already proven confusing to some users, and would add complexity to the code as well.

As for the situation you've described in the Discuss post linked above:

It worked well except for the last index automatically created by the rollover aliased, which can contain very few documents. I know it can be fine because it is the only one with 8 shards for the ILM managed index, but I'd like to avoid having indices with several shards, because the same policy could be applied to numerous other indices, so I'd like to limit the number of shards in the cluster.

We have a planned improvement to ILM to allow manually rolling over indices without causing errors in ILM (#44175). Once we've implemented this, when you've determined an index pattern is "complete", you could manually roll over the index, which would allow that index to continue through its lifecycle, then manually delete the new, empty index. As for the concerns about the same policy being applied to other indices, we recommend creating separate policies if you have indices with different requirements - ILM policies are relatively cheap, so you shouldn't worry about having multiple policies.

During the discussion, a more generally applicable use case came up: ILM-managed indices with max_age conditions which have no indexing going on, potentially for long periods of time. This can be the case if a Beat is set up, but then never run. We opened #46161, describing a feature which would allow indices to progress through their lifecycle immediately if they contain zero documents. This may be helpful in the above case as well.

I'm going to close this issue, as we do not intend to implement this feature at the moment. However, if anyone finds this and has this or another use case that this feature would address, please comment here.

[peterpramb]

Just a slightly different use case for ingest failures:

• ingest-failures policy

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_size": "1gb"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

• Indices

health status index                  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   ingest-failures-000001 THOGb_NRT8GfJaU2_CwAVQ   1   1         15            0      143kb         53.7kb
green  open   ingest-failures-000002 j87TKyYzSma3sGpr5-CWmw   1   1     397541            0    377.9mb        189.1mb
green  open   ingest-failures-000003 G7KcHrq0Tvu3X5DOIxAzCQ   1   1          0            0       566b           283b
green  open   ingest-failures-000004 Qo2k_dyAT3CS7Ke2nrYObQ   1   1          0            0       566b           283b

So this might eventually cause issues in a long-running cluster with a very short max_age setting and a poorly choosen index suffix number (is an overflow there allowed?), when one does not expect empty indices to roll over at all.

Beside that it's probably more an cosmetic issue...

[nemphys]

+1 on this one, there are cases when data streams become obsolete (with no new data) but should not be deleted (keep in warm/cold/frozen state forever); this leads to infinite new (empty) indices in the data stream.