[RollupV2]: make RollupAction available and improve some features

[weizijun]

Background

1. Rollup is very useful in TSDB case. TSDB can use rollup to do timestamp downsample.
2. The Rollup refactoring is under development at Refactor rollups meta (AKA Rollup V2) #42720, and it seems to be blocked.

I know rollups is a bit of a complicated beast of a project. Yes, there may be a lot of design details to discuss. But downsampling is very useful in our use case. So I'm continuing to develop rollup refactoring.

Detail

I think the RollupAction is the basement of the rollup refactoring. So I do some improvement to make the API available to use and add some feature in the RollupAction API.

Here's a list of what I've done:

1. Fix RollupAction API bugs:

• Fix when write sparse metric column, it will cause write exception.
• Fix when read sparse group column, it will cause NPE exception.
• Fix when date_histogram field spare, calculate range query exception.
• Fix when source index not found, it will case NPE exception.
• Fix when rollup index is exists, it will continue to rollup until finished rollup, cause resize exception.
• Fix when origin index's shard is more than 1, RollupIndexerAction get shard list error.

2. Add some features

• Rollup group, metrics support wildcard fields.
• Support rollup index reuse the origin index's settings.
• Support get rollup tasks status, and support cancel rollup tasks.
• Support rollup group terms configured _tsid field.
• Support rollup time_series index in the new execution mode(Add a flag to control in index order execution mode to aggregations #82129). It greatly improves the performance of rollup tasks.
• Logging optimize: log the execution cost, optimize index failed log, and so on.
• Rollup ilm policy index optimize, remove the ilm police in tmp index, and add back in the resize settings
• Add many tests, to make the code coverage more than 90%.

Since this is the first PR of my rollup feature, I won't change the code that might affect other features.

Some features are ready in our internal repository. Here is the TODO list in the next PRs:

• Support rollup aggregate_metric_double type metrics, as rollup index use aggregate_metric_double to store metric type, so we can't rollup a rollupe index.
• Support search rollup index, now when rollup a data stream index, the rollup index will also add to data stream's concrete indices, It will cause error search result. And the search rollup function will select the best indices for a search DSL that match the rollup config.

There are a few more things to discuss:

• If the origin index must be a read-only index? if yes, it will add the check logic.
• Now I add the logic, that rollup a standard index to time_series, as the rollup index is made of timestamp\dimensions\metrics. Is it ok to do this? And how to deal with the start_time and end_time, now I add the largest bounds as the standard index has no bounds.
• Now, there is only one thread per node to execute rollup task. In some node with many cpus, it may slow down rollup tasks. But adding more threads to execute rollup task will take more cpu costs, Maybe the thread size can be adapted to the node.processors setting.
• How to handle the failed index docs, record the exception or failed the rollup task?
• How to set the rollup index number_of_shards? Now the shard count of rollup index is the same with origin index. But in many case, the size of rollup index is much small than the origin index. Is it necessary to set the custom rollup index number_of_shards.
• How to do anomaly testing in Elasticsearch automatically? The node shut down case? The disk grows to 100% case? The rolluping shard reallocate to other node case?

And about the Rollup performance, In this PR, I don't change the process of unsorted index. But I think it will have some performance improvement. I will add a todo task to think more about it.

[weizijun]

In this PR, I don't add the AggregateDoubleMetricField child class, as I will modify some code of AggregateDoubleMetric, I will add the class, and support aggregate_metric_double rollup in the next PR.

[weizijun]

@elasticmachine update branch

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[csoulios]

@weizijun thank you for submitting this PR. It adds a lot of functionality and fixes many issues.

For a start, I would like to say that the existing RollupAction implementation is
more a POC than a robust production-ready code. We took the decision to stop
working on this code until the TSDB project has taken shape, so that we
can make the most of the TSDB features.

Time series indices present all those characteristics that allow us to optimise downsampling for performance as well as user experience. Therefore, for the first release of rollups, we intend to support downsampling only for time series indices. Implementing downsampling for non time series indices is not in our immediate plans.

Rollup configuration

So, starting with the RollupAction, we plan to simplify the rollup configuration
by reducing the input parameters to the bare minimum, now that we are able to
infer most of them from the source index mapping.

For example, for time series indices the the timestamp field is always called
@timestamp. So, we must only add the downsampling interval and the time zone.

Also, for downsampling we are not allowed to modify the dimensions and and we must keep _tsid as is. So, the groups part of the rollup configuration will be shrinked to only the downsampling interval.

In the same way, we can remove the metrics part of the configuration by
extracting the metric fields from the index mapping.

This would simplify the Rollup API for downsampling to something like
the following:

POST /<index>/_rollup/<rollup_index>
{
    "calendar_interval": "1M",
    "time_zone": "UTC" // Default value is UTC
}

Performance

Regarding performance, we plan to use the TimeSeriesMetricsService (#82129) so that we iterate over the time series index. This allows us to leverage the fact that
the index is sorted per _tsid

Finally, this PR contains so many fixes, improvements and changes that I would like us
to add in smaller chunks

[weizijun]

@csoulios, Thank you so much for your answer.

Time series indices present all those characteristics that allow us to optimise downsampling for performance as well as user experience. Therefore, for the first release of rollups, we intend to support downsampling only for time series indices. Implementing downsampling for non time series indices is not in our immediate plans.

yeah, I do the performance improvement for time_series indices in TimeSeriesRollupShardIndexer, it use TimeSeriesIndexSearcher to iterator all doc in order. It can great faster than the normal indices. And it's okay about use TimeSeriesMetricsService to improve the performance, we can have more discussion.

This would simplify the Rollup API for downsampling to something like the following:

POST /<index>/_rollup/<rollup_index>
{
    "calendar_interval": "1M",
    "time_zone": "UTC" // Default value is UTC
}

About Rollup configuration, ​I think TSDB downsampling is a special case of Rollup. Downsampling terms group has only _tsid, and metrics are all fields with time_series_metric configured, and the date_histogram field is @timestamp. Users only need to configure interval and time_zone.

I prefer downsampling is a new API, like:

POST /<index>/_downsample/<rollup_index>
{
    "fixed_interval": "10m"
}

it equals the API:

POST /<index>/_rollup/<rollup_index>
{
  "groups": {
    "date_histogram": {
      "field": "@timestamp",
      "fixed_interval": "10m"
    },
    "terms": {
      "fields": [
        "_tsid"
      ]
    }
  },
  "metrics": [
    {
      "field": "m1",
      "metrics": [
        "min",
        "max",
        "sum",
        "value_count"
      ]
    },
    {
      "field": "m2",
      "metrics": [
        "min",
        "max",
        "sum",
        "value_count"
      ]
    }
  ]
}

The new DownsampleAction can reuse many code with RollupAction.
We Alibaba Cloud Elasticsearch has been practice to use RollupAction downsample TSDB indices. User only configure the interval settings, our service generate the RollupAction config, it's easy to use.

The code of this PR is our practice in Alibaba Cloud Elasticsearch, so I add many feature. I hope we can have more deep discuss, and to see how to implement the rollup and downsample feature.

[csoulios]

Closing this PR because it overlaps with the work at PR #85708

@weizijun and I have discussed about the dowsampling implementation and the path forward.
Thank you for this contribution.