Integrate rollup search with _search endpoint

[csoulios]

Currently searching in rollups is facilitated by a specific endpoint named _rollup_search. For seamless integration of rollups we should remove this endpoint and implement its functionality in the _search endpoint.

When running aggregations using the existing _search endpoint we must implement the following:

• Merge rollup buckets with regular buckets.
• Ensure that partial buckets can merge across shards (e.g. when rollover is not on a clean interval).
• Modify existing aggregators to pull doc_count from rollup indices, rather than just incrementing. This is needed so a "rollup metric" can provide doc counts for avg and bucketing aggregations, by allowing the aggregator framework to treat a single document as if it were multiple docs.

Producing rollups is discussed in #48003

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Rollup)

[timroes]

💯 This would be very awesome and ease a lot of code in Kibana, if we would have the integration in the regular _search endpoint.

[lizozom]

@csoulios is there a target version for this change?

Anyway, please ping @elastic/kibana-app-arch when this is being implemented, so we can go ahead an implement elastic/kibana#64873. 🙏

[tangpf877]

Is there any news about rollup? I am wandering when _rollup_search endpoint will be removed, and what the ILM side will be like in the future? Does 'Create an ILM rollup action' means rollup will proceed automatically and we needn't start an rollup job? Thanks for reply

[csoulios]

Hi @benjam1n77, rollups refactoring is a task we are actively working on. You can track progress and related discussions in this ticket: #42720

When it comes to ILM and rollups, the goal is to stop having rollups as continues jobs. Instead rollup will be an action in the ILM configuration. See the relevant ticket for more: #48003

[wchaparro]

With the 8.7 release of Elasticsearch, we have made a new downsampling capability associated with the new time series datastreams functionality generally available (GA). This capability was in tech preview in ILM since 8.5. Downsampling provides a method to reduce the footprint of your time series data by storing it at reduced granularity. The downsampling process rolls up documents within a fixed time interval into a single summary document. Each summary document includes statistical representations of the original data: the min, max, sum, value_count, and average for each metric. Data stream time series dimensions are stored unchanged.

Downsampling is superior to rollup because:

• Downsampled indices are searched through the _search API
• It is possible to query multiple downsampled indices together with raw data indices
• The pre-aggregation is based on the metrics and time series definitions in the index mapping so very little configuration is required (i.e. much easier to add new time serieses)
• Downsampling is managed as an action in ILM
• It is possible to downsample a downsampled index, and reduce granularity as the index ages
• The performance of the pre-aggregation process is superior in downsampling, as it builds on the time_series index mode infrastructure

Because of the introduction of this new capability, we are deprecating the rollups functionality, which never left the Tech Preview/Experimental status, in favor of downsampling and thus we are closing this issue. We encourage you to migrate your solution to downsampling and take advantage of the new TSDB functionality.