Skip to content

Commit d54f00a

Browse files
polyfractalpolyfractal
authored
[Rollup] Add new capabilities endpoint for concrete rollup indices (#32111)
This introduces a new GetRollupIndexCaps API which allows the user to retrieve rollup capabilities of a specific rollup index (or index pattern). This is distinct from the existing RollupCaps endpoint. - Multiple jobs can be stored in multiple indices and point to a single target data index pattern (logstash-*). The existing API finds capabilities/config of all jobs matching that data index pattern. - One rollup index can hold data from multiple jobs, targeting multiple data index patterns. This new API finds the capabilities based on the concrete rollup indices.
1 parent df36d1c commit d54f00a

File tree

12 files changed

+1071
-17
lines changed

12 files changed

+1071
-17
lines changed

‎x-pack/docs/en/rest-api/rollup/rollup-index-caps.asciidoc

+161
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
[role="xpack"]
2+
[[rollup-get-rollup-index-caps]]
3+
=== Get Rollup Index Capabilities
4+
++++
5+
<titleabbrev>Get Rollup Index Caps</titleabbrev>
6+
++++
7+
8+
experimental[]
9+
10+
This API returns the rollup capabilities of all jobs inside of a rollup index (e.g. the index where rollup data is stored).
11+
A single rollup index may store the data for multiple rollup jobs, and may have a variety of capabilities depending on those jobs.
12+
13+
This API will allow you to determine:
14+
15+
1. What jobs are stored in an index (or indices specified via a pattern)?
16+
2. What target indices were rolled up, what fields were used in those rollups and what aggregations can be performed on each job?
17+
18+
==== Request
19+
20+
`GET {index}/_xpack/rollup/data`
21+
22+
//===== Description
23+
24+
==== Path Parameters
25+
26+
`index`::
27+
(string) Index or index-pattern of concrete rollup indices to check for capabilities.
28+
29+
30+
31+
==== Request Body
32+
33+
There is no request body for the Get Jobs API.
34+
35+
==== Authorization
36+
37+
You must have `monitor`, `monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API.
38+
For more information, see
39+
{xpack-ref}/security-privileges.html[Security Privileges].
40+
41+
==== Examples
42+
43+
Imagine we have an index named `sensor-1` full of raw data. We know that the data will grow over time, so there
44+
will be a `sensor-2`, `sensor-3`, etc. Let's create a Rollup job, which stores it's data in `sensor_rollup`:
45+
46+
[source,js]
47+
--------------------------------------------------
48+
PUT _xpack/rollup/job/sensor
49+
{
50+
"index_pattern": "sensor-*",
51+
"rollup_index": "sensor_rollup",
52+
"cron": "*/30 * * * * ?",
53+
"page_size" :1000,
54+
"groups" : {
55+
"date_histogram": {
56+
"field": "timestamp",
57+
"interval": "1h",
58+
"delay": "7d"
59+
},
60+
"terms": {
61+
"fields": ["node"]
62+
}
63+
},
64+
"metrics": [
65+
{
66+
"field": "temperature",
67+
"metrics": ["min", "max", "sum"]
68+
},
69+
{
70+
"field": "voltage",
71+
"metrics": ["avg"]
72+
}
73+
]
74+
}
75+
--------------------------------------------------
76+
// CONSOLE
77+
// TEST[setup:sensor_index]
78+
79+
If at a later date, we'd like to determine what jobs and capabilities were stored in the `sensor_rollup` index, we can use the Get Rollup
80+
Index API:
81+
82+
[source,js]
83+
--------------------------------------------------
84+
GET /sensor_rollup/_xpack/rollup/data
85+
--------------------------------------------------
86+
// CONSOLE
87+
// TEST[continued]
88+
89+
Note how we are requesting the concrete rollup index name (`sensor_rollup`) as the first part of the URL.
90+
This will yield the following response:
91+
92+
[source,js]
93+
----
94+
{
95+
"sensor_rollup" : {
96+
"rollup_jobs" : [
97+
{
98+
"job_id" : "sensor",
99+
"rollup_index" : "sensor_rollup",
100+
"index_pattern" : "sensor-*",
101+
"fields" : {
102+
"node" : [
103+
{
104+
"agg" : "terms"
105+
}
106+
],
107+
"temperature" : [
108+
{
109+
"agg" : "min"
110+
},
111+
{
112+
"agg" : "max"
113+
},
114+
{
115+
"agg" : "sum"
116+
}
117+
],
118+
"timestamp" : [
119+
{
120+
"agg" : "date_histogram",
121+
"time_zone" : "UTC",
122+
"interval" : "1h",
123+
"delay": "7d"
124+
}
125+
],
126+
"voltage" : [
127+
{
128+
"agg" : "avg"
129+
}
130+
]
131+
}
132+
}
133+
]
134+
}
135+
}
136+
----
137+
// TESTRESPONSE
138+
139+
140+
The response that is returned contains information that is similar to the original Rollup configuration, but formatted
141+
differently. First, there are some house-keeping details: the Rollup job's ID, the index that holds the rolled data,
142+
the index pattern that the job was targeting.
143+
144+
Next it shows a list of fields that contain data eligible for rollup searches. Here we see four fields: `node`, `temperature`,
145+
`timestamp` and `voltage`. Each of these fields list the aggregations that are possible. For example, you can use a min, max
146+
or sum aggregation on the `temperature` field, but only a `date_histogram` on `timestamp`.
147+
148+
Note that the `rollup_jobs` element is an array; there can be multiple, independent jobs configured for a single index
149+
or index pattern. Each of these jobs may have different configurations, so the API returns a list of all the various
150+
configurations available.
151+
152+
153+
Like other APIs that interact with indices, you can specify index patterns instead of explicit indices:
154+
155+
[source,js]
156+
--------------------------------------------------
157+
GET /*_rollup/_xpack/rollup/data
158+
--------------------------------------------------
159+
// CONSOLE
160+
// TEST[continued]
161+

‎x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rollup/action/GetRollupCapsAction.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ public void writeTo(StreamOutput out) throws IOException {
145145
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
146146
builder.startObject();
147147
for (Map.Entry<String, RollableIndexCaps> entry : jobs.entrySet()) {
148-
entry.getValue().toXContent(builder, params);
148+
entry.getValue().toXContent(builder, params);
149149
}
150150
builder.endObject();
151151
return builder;

‎x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rollup/action/GetRollupIndexCapsAction.java

+201
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License;
4+
* you may not use this file except in compliance with the Elastic License.
5+
*/
6+
package org.elasticsearch.xpack.core.rollup.action;
7+
8+
9+
import org.elasticsearch.action.Action;
10+
import org.elasticsearch.action.ActionRequest;
11+
import org.elasticsearch.action.ActionRequestBuilder;
12+
import org.elasticsearch.action.ActionRequestValidationException;
13+
import org.elasticsearch.action.ActionResponse;
14+
import org.elasticsearch.action.IndicesRequest;
15+
import org.elasticsearch.action.support.IndicesOptions;
16+
import org.elasticsearch.client.ElasticsearchClient;
17+
import org.elasticsearch.common.ParseField;
18+
import org.elasticsearch.common.Strings;
19+
import org.elasticsearch.common.io.stream.StreamInput;
20+
import org.elasticsearch.common.io.stream.StreamOutput;
21+
import org.elasticsearch.common.io.stream.Writeable;
22+
import org.elasticsearch.common.xcontent.ToXContent;
23+
import org.elasticsearch.common.xcontent.ToXContentObject;
24+
import org.elasticsearch.common.xcontent.XContentBuilder;
25+
import org.elasticsearch.xpack.core.rollup.RollupField;
26+
27+
import java.io.IOException;
28+
import java.util.Arrays;
29+
import java.util.Collections;
30+
import java.util.Map;
31+
import java.util.Objects;
32+
33+
public class GetRollupIndexCapsAction extends Action<GetRollupIndexCapsAction.Request,
34+
GetRollupIndexCapsAction.Response, GetRollupIndexCapsAction.RequestBuilder> {
35+
36+
public static final GetRollupIndexCapsAction INSTANCE = new GetRollupIndexCapsAction();
37+
public static final String NAME = "indices:data/read/xpack/rollup/get/index/caps";
38+
public static final ParseField CONFIG = new ParseField("config");
39+
public static final ParseField STATUS = new ParseField("status");
40+
private static final ParseField INDICES_OPTIONS = new ParseField("indices_options");
41+
42+
private GetRollupIndexCapsAction() {
43+
super(NAME);
44+
}
45+
46+
@Override
47+
public Response newResponse() {
48+
return new Response();
49+
}
50+
51+
@Override
52+
public RequestBuilder newRequestBuilder(ElasticsearchClient client) {
53+
return new RequestBuilder(client, INSTANCE);
54+
}
55+
56+
public static class Request extends ActionRequest implements IndicesRequest.Replaceable, ToXContent {
57+
private String[] indices;
58+
private IndicesOptions options;
59+
60+
public Request(String[] indices) {
61+
this(indices, IndicesOptions.STRICT_EXPAND_OPEN_FORBID_CLOSED);
62+
}
63+
64+
public Request(String[] indices, IndicesOptions options) {
65+
this.indices = indices;
66+
this.options = options;
67+
}
68+
69+
public Request() {}
70+
71+
@Override
72+
public IndicesOptions indicesOptions() {
73+
return options;
74+
}
75+
76+
@Override
77+
public String[] indices() {
78+
return indices;
79+
}
80+
81+
@Override
82+
public IndicesRequest indices(String... indices) {
83+
Objects.requireNonNull(indices, "indices must not be null");
84+
for (String index : indices) {
85+
Objects.requireNonNull(index, "index must not be null");
86+
}
87+
this.indices = indices;
88+
return this;
89+
}
90+
91+
@Override
92+
public void readFrom(StreamInput in) throws IOException {
93+
super.readFrom(in);
94+
this.indices = in.readStringArray();
95+
this.options = IndicesOptions.readIndicesOptions(in);
96+
}
97+
98+
@Override
99+
public void writeTo(StreamOutput out) throws IOException {
100+
super.writeTo(out);
101+
out.writeStringArray(indices);
102+
options.writeIndicesOptions(out);
103+
}
104+
105+
@Override
106+
public ActionRequestValidationException validate() {
107+
return null;
108+
}
109+
110+
@Override
111+
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
112+
builder.array(RollupField.ID.getPreferredName(), indices);
113+
builder.field(INDICES_OPTIONS.getPreferredName(), options);
114+
return builder;
115+
}
116+
117+
@Override
118+
public int hashCode() {
119+
return Objects.hash(Arrays.hashCode(indices), options);
120+
}
121+
122+
@Override
123+
public boolean equals(Object obj) {
124+
if (obj == null) {
125+
return false;
126+
}
127+
if (getClass() != obj.getClass()) {
128+
return false;
129+
}
130+
Request other = (Request) obj;
131+
return Arrays.equals(indices, other.indices)
132+
&& Objects.equals(options, other.options);
133+
}
134+
}
135+
136+
public static class RequestBuilder extends ActionRequestBuilder<Request, Response, RequestBuilder> {
137+
138+
protected RequestBuilder(ElasticsearchClient client, GetRollupIndexCapsAction action) {
139+
super(client, action, new Request());
140+
}
141+
}
142+
143+
public static class Response extends ActionResponse implements Writeable, ToXContentObject {
144+
145+
private Map<String, RollableIndexCaps> jobs = Collections.emptyMap();
146+
147+
public Response() {
148+
149+
}
150+
151+
public Response(Map<String, RollableIndexCaps> jobs) {
152+
this.jobs = Objects.requireNonNull(jobs);
153+
}
154+
155+
Response(StreamInput in) throws IOException {
156+
jobs = in.readMap(StreamInput::readString, RollableIndexCaps::new);
157+
}
158+
159+
public Map<String, RollableIndexCaps> getJobs() {
160+
return jobs;
161+
}
162+
163+
@Override
164+
public void writeTo(StreamOutput out) throws IOException {
165+
super.writeTo(out);
166+
out.writeMap(jobs, StreamOutput::writeString, (out1, value) -> value.writeTo(out1));
167+
}
168+
169+
@Override
170+
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
171+
builder.startObject();
172+
for (Map.Entry<String, RollableIndexCaps> entry : jobs.entrySet()) {
173+
entry.getValue().toXContent(builder, params);
174+
}
175+
builder.endObject();
176+
return builder;
177+
}
178+
179+
@Override
180+
public int hashCode() {
181+
return Objects.hash(jobs);
182+
}
183+
184+
@Override
185+
public boolean equals(Object obj) {
186+
if (obj == null) {
187+
return false;
188+
}
189+
if (getClass() != obj.getClass()) {
190+
return false;
191+
}
192+
Response other = (Response) obj;
193+
return Objects.equals(jobs, other.jobs);
194+
}
195+
196+
@Override
197+
public final String toString() {
198+
return Strings.toString(this);
199+
}
200+
}
201+
}

0 commit comments

Comments
 (0)