Skip to content

Commit d4262de

Browse files
authored
[Docs] All Rollup docs experimental, agg limitations, clarify DeleteJob (#31299)
- All rollup pages should be marked as experimental instead of just the top page - While the job config docs state which aggregations are allowed, adding a section which specifically details this in one place is more convenient for the user - Add a clarification that the DeleteJob API does not delete the rollup data, just the rollup job.
1 parent 664903a commit d4262de

15 files changed

+81
-3
lines changed

x-pack/docs/en/rest-api/rollup/delete-job.asciidoc

+28
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,37 @@
55
<titleabbrev>Delete Job</titleabbrev>
66
++++
77

8+
experimental[]
9+
810
This API deletes an existing rollup job. The job can be started or stopped, in both cases it will be deleted. Attempting
911
to delete a non-existing job will throw an exception
1012

13+
.Deleting the job does not delete rolled up data
14+
**********************************
15+
When a job is deleted, that only removes the process that is actively monitoring and rolling up data.
16+
It does not delete any previously rolled up data. This is by design; a user may wish to roll up a static dataset. Because
17+
the dataset is static, once it has been fully rolled up there is no need to keep the indexing Rollup job around (as there
18+
will be no new data). So the job may be deleted, leaving behind the rolled up data for analysis.
19+
20+
If you wish to also remove the rollup data, and the rollup index only contains the data for a single job, you can simply
21+
delete the whole rollup index. If the rollup index stores data from several jobs, you must issue a Delete-By-Query that
22+
targets the Rollup job's ID in the rollup index:
23+
24+
25+
[source,js]
26+
--------------------------------------------------
27+
POST my_rollup_index/_delete_by_query
28+
{
29+
"query": {
30+
"term": {
31+
"_rollup.id": "the_rollup_job_id"
32+
}
33+
}
34+
}
35+
--------------------------------------------------
36+
// NOTCONSOLE
37+
38+
**********************************
1139
==== Request
1240

1341
`DELETE _xpack/rollup/job/<job_id>`

x-pack/docs/en/rest-api/rollup/get-job.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
<titleabbrev>Get Job</titleabbrev>
66
++++
77

8+
experimental[]
9+
810
This API returns the configuration, stats and status of rollup jobs. The API can return the details for a single job,
911
or for all jobs.
1012

x-pack/docs/en/rest-api/rollup/put-job.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
<titleabbrev>Create Job</titleabbrev>
66
++++
77

8+
experimental[]
9+
810
This API enables you to create a rollup job. The job will be created in a `STOPPED` state, and must be
911
started with the <<rollup-start-job,Start Job API>>.
1012

x-pack/docs/en/rest-api/rollup/rollup-caps.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
<titleabbrev>Get Rollup Caps</titleabbrev>
66
++++
77

8+
experimental[]
9+
810
This API returns the rollup capabilities that have been configured for an index or index pattern. This API is useful
911
because a rollup job is often configured to rollup only a subset of fields from the source index. Furthermore, only
1012
certain aggregations can be configured for various fields, leading to a limited subset of functionality depending on

x-pack/docs/en/rest-api/rollup/rollup-job-config.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22
[[rollup-job-config]]
33
=== Rollup Job Configuration
44

5+
experimental[]
6+
57
The Rollup Job Configuration contains all the details about how the rollup job should run, when it indexes documents,
68
and what future queries will be able to execute against the rollup index.
79

x-pack/docs/en/rest-api/rollup/rollup-search.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
<titleabbrev>Rollup Search</titleabbrev>
66
++++
77

8+
experimental[]
9+
810
The Rollup Search endpoint allows searching rolled-up data using the standard query DSL. The Rollup Search endpoint
911
is needed because, internally, rolled-up documents utilize a different document structure than the original data. The
1012
Rollup Search endpoint rewrites standard query DSL into a format that matches the rollup documents, then takes the response

x-pack/docs/en/rest-api/rollup/start-job.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
<titleabbrev>Start Job</titleabbrev>
66
++++
77

8+
experimental[]
9+
810
This API starts an existing, stopped rollup job. If the job does not exist an exception will be thrown.
911
Starting an already started job has no action.
1012

x-pack/docs/en/rest-api/rollup/stop-job.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
<titleabbrev>Stop Job</titleabbrev>
66
++++
77

8+
experimental[]
9+
810
This API stops an existing, started rollup job. If the job does not exist an exception will be thrown.
911
Stopping an already stopped job has no action.
1012

x-pack/docs/en/rollup/api-quickref.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
[[rollup-api-quickref]]
22
== API Quick Reference
33

4+
experimental[]
5+
46
Most {rollup} endpoints have the following base:
57

68
[source,js]

x-pack/docs/en/rollup/index.asciidoc

+3-1
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@ for analysis, but at a fraction of the storage cost of raw data.
1818
* <<rollup-getting-started,Getting Started>>
1919
* <<rollup-api-quickref, API Quick Reference>>
2020
* <<rollup-understanding-groups,Understanding Rollup Grouping>>
21-
* <<rollup-search-limitations,Limitations of Rollup Search>>
21+
* <<rollup-agg-limitations,Rollup aggregation limitations>>
22+
* <<rollup-search-limitations,Rollup Search limitations>>
2223

2324

2425
--
@@ -27,4 +28,5 @@ include::overview.asciidoc[]
2728
include::api-quickref.asciidoc[]
2829
include::rollup-getting-started.asciidoc[]
2930
include::understanding-groups.asciidoc[]
31+
include::rollup-agg-limitations.asciidoc[]
3032
include::rollup-search-limitations.asciidoc[]

x-pack/docs/en/rollup/overview.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
[[rollup-overview]]
22
== Overview
33

4+
experimental[]
5+
46
Time-based data (documents that are predominantly identified by their timestamp) often have associated retention policies
57
to manage data growth. For example, your system may be generating 500,000 documents every second. That will generate
68
43 million documents per day, and nearly 16 billion documents a year.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
[[rollup-agg-limitations]]
2+
== Rollup Aggregation Limitations
3+
4+
experimental[]
5+
6+
There are some limitations to how fields can be rolled up / aggregated. This page highlights the major limitations so that
7+
you are aware of them.
8+
9+
[float]
10+
=== Limited aggregation components
11+
12+
The Rollup functionality allows fields to be grouped with the following aggregations:
13+
14+
- Date Histogram aggregation
15+
- Histogram aggregation
16+
- Terms aggregation
17+
18+
And the following metrics are allowed to be specified for numeric fields:
19+
20+
- Min aggregation
21+
- Max aggregation
22+
- Sum aggregation
23+
- Average aggregation
24+
- Value Count aggregation

x-pack/docs/en/rollup/rollup-getting-started.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
[[rollup-getting-started]]
22
== Getting Started
33

4+
experimental[]
5+
46
To use the Rollup feature, you need to create one or more "Rollup Jobs". These jobs run continuously in the background
57
and rollup the index or indices that you specify, placing the rolled documents in a secondary index (also of your choosing).
68

x-pack/docs/en/rollup/rollup-search-limitations.asciidoc

+4-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
[[rollup-search-limitations]]
22
== Rollup Search Limitations
33

4+
experimental[]
5+
46
While we feel the Rollup function is extremely flexible, the nature of summarizing data means there will be some limitations. Once
57
live data is thrown away, you will always lose some flexibility.
68

@@ -100,8 +102,8 @@ The Rollup functionality allows `query`'s in the search request, but with a limi
100102
- MatchAll Query
101103
- Any compound query (Boolean, Boosting, ConstantScore, etc)
102104

103-
Furthermore, these queries can only use fields that were also saved in the rollup job. If you wish to filter on a keyword `hostname` field,
104-
that field must have been configured in the rollup job under a `terms` grouping.
105+
Furthermore, these queries can only use fields that were also saved in the rollup job as a `group`.
106+
If you wish to filter on a keyword `hostname` field, that field must have been configured in the rollup job under a `terms` grouping.
105107

106108
If you attempt to use an unsupported query, or the query references a field that wasn't configured in the rollup job, an exception will be
107109
thrown. We expect the list of support queries to grow over time as more are implemented.

x-pack/docs/en/rollup/understanding-groups.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
[[rollup-understanding-groups]]
22
== Understanding Groups
33

4+
experimental[]
5+
46
To preserve flexibility, Rollup Jobs are defined based on how future queries may need to use the data. Traditionally, systems force
57
the admin to make decisions about what metrics to rollup and on what interval. E.g. The average of `cpu_time` on an hourly basis. This
68
is limiting; if, at a future date, the admin wishes to see the average of `cpu_time` on an hourly basis _and partitioned by `host_name`_,

0 commit comments

Comments
 (0)