Skip to content

Commit 47706b5

Browse files
authored
Add index mode to get data stream API (#122486)
This commit adds the `index_mode` for both the data stream and each backing index to the output of `GET /_data_stream`. An example looks like: ``` { "data_streams" : [ { "name" : "foo-things", "indices" : [ { "index_name" : ".ds-foo-things-2025.02.13-000001", ... "index_mode" : "standard" } ], ... "index_mode" : "standard" }, { "name" : "logs-foo-bar", "indices" : [ { "index_name" : ".ds-logs-foo-bar-2025.02.13-000001", ... "index_mode" : "logsdb" }, { "index_name" : ".ds-logs-foo-bar-2025.02.13-000002", ... "index_mode" : "logsdb" } ], ... "index_mode" : "logsdb", } ] } ```
1 parent 2fa6651 commit 47706b5

File tree

11 files changed

+585
-26
lines changed

11 files changed

+585
-26
lines changed

docs/changelog/122486.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 122486
2+
summary: Add index mode to get data stream API
3+
area: Data streams
4+
type: enhancement
5+
issues: []
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,376 @@
1+
[role="xpack"]
2+
[[indices-get-data-stream]]
3+
=== Get data stream API
4+
++++
5+
<titleabbrev>Get data stream</titleabbrev>
6+
++++
7+
8+
.New API reference
9+
[sidebar]
10+
--
11+
For the most up-to-date API details, refer to {api-es}/group/endpoint-data-stream[Data stream APIs].
12+
--
13+
14+
Retrieves information about one or more <<data-streams,data streams>>.
15+
See <<get-info-about-data-stream>>.
16+
17+
////
18+
[source,console]
19+
----
20+
PUT /_ilm/policy/my-lifecycle-policy
21+
{
22+
"policy": {
23+
"phases": {
24+
"hot": {
25+
"actions": {
26+
"rollover": {
27+
"max_primary_shard_size": "25GB"
28+
}
29+
}
30+
},
31+
"delete": {
32+
"min_age": "30d",
33+
"actions": {
34+
"delete": {}
35+
}
36+
}
37+
}
38+
}
39+
}
40+
41+
PUT /_index_template/my-index-template
42+
{
43+
"index_patterns": [ "my-data-stream*" ],
44+
"data_stream": {},
45+
"template": {
46+
"settings": {
47+
"index.lifecycle.name": "my-lifecycle-policy"
48+
}
49+
},
50+
"_meta": {
51+
"my-meta-field": "foo"
52+
}
53+
}
54+
55+
PUT /_data_stream/my-data-stream
56+
57+
POST /my-data-stream/_rollover
58+
59+
PUT /_data_stream/my-data-stream-two
60+
61+
DELETE /_data_stream/my-data-stream*/_lifecycle
62+
----
63+
// TESTSETUP
64+
////
65+
66+
////
67+
[source,console]
68+
----
69+
DELETE /_data_stream/*
70+
DELETE /_index_template/*
71+
DELETE /_ilm/policy/my-lifecycle-policy
72+
----
73+
// TEARDOWN
74+
////
75+
76+
[source,console]
77+
----
78+
GET /_data_stream/my-data-stream
79+
----
80+
81+
[[get-data-stream-api-request]]
82+
==== {api-request-title}
83+
84+
`GET /_data_stream/<data-stream>`
85+
86+
[[get-data-stream-api-prereqs]]
87+
==== {api-prereq-title}
88+
89+
* If the {es} {security-features} are enabled, you must have the
90+
`view_index_metadata` or `manage` <<privileges-list-indices,index privilege>>
91+
for the data stream.
92+
93+
[[get-data-stream-api-path-params]]
94+
==== {api-path-parms-title}
95+
96+
`<data-stream>`::
97+
(Optional, string)
98+
Comma-separated list of data stream names used to limit the request. Wildcard
99+
(`*`) expressions are supported. If omitted, all data streams will be
100+
returned.
101+
102+
[role="child_attributes"]
103+
[[get-data-stream-api-query-parms]]
104+
==== {api-query-parms-title}
105+
106+
include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=ds-expand-wildcards]
107+
+
108+
Defaults to `open`.
109+
110+
`include_defaults`::
111+
(Optional, Boolean) Functionality in preview:[]. If `true`, return all default settings in the response.
112+
Defaults to `false`.
113+
114+
`verbose`::
115+
(Optional, Boolean). If `true`, Returns the `maximum_timestamp` corresponding to the `@timestamp` field for documents in the data stream.
116+
Defaults to `false`.
117+
118+
[role="child_attributes"]
119+
[[get-data-stream-api-response-body]]
120+
==== {api-response-body-title}
121+
122+
`data_streams`::
123+
(array of objects)
124+
Contains information about retrieved data streams.
125+
+
126+
.Properties of objects in `data_streams`
127+
[%collapsible%open]
128+
====
129+
`name`::
130+
(string)
131+
Name of the data stream.
132+
133+
`timestamp_field`::
134+
(object)
135+
Contains information about the data stream's `@timestamp` field.
136+
+
137+
.Properties of `timestamp_field`
138+
[%collapsible%open]
139+
=====
140+
`name`::
141+
(string)
142+
Name of the data stream's timestamp field, which must be `@timestamp`. The
143+
`@timestamp` field must be included in every document indexed to the data
144+
stream.
145+
=====
146+
147+
`indices`::
148+
(array of objects)
149+
Array of objects containing information about the data stream's backing
150+
indices.
151+
+
152+
The last item in this array contains information about the stream's current
153+
<<data-stream-write-index,write index>>.
154+
+
155+
.Properties of `indices` objects
156+
[%collapsible%open]
157+
=====
158+
`index_name`::
159+
(string)
160+
Name of the backing index. For naming conventions, see
161+
<<data-streams-generation>>.
162+
163+
`index_uuid`::
164+
(string)
165+
Universally unique identifier (UUID) for the index.
166+
167+
`prefer_ilm`::
168+
(boolean)
169+
Functionality in preview:[]. Indicates if this index is configured to prefer {ilm}
170+
when both {ilm-cap} and <<data-stream-lifecycle, Data stream lifecycle>> are configured to
171+
manage this index.
172+
173+
`managed_by`::
174+
(string)
175+
Functionality in preview:[]. Indicates the system that managed this index.
176+
=====
177+
178+
`generation`::
179+
(integer)
180+
Current <<data-streams-generation,generation>> for the data stream. This number
181+
acts as a cumulative count of the stream's rollovers, starting at `1`.
182+
183+
`_meta`::
184+
(object)
185+
Custom metadata for the stream, copied from the `_meta` object of the
186+
stream's matching <<create-index-template,index template>>. If empty,
187+
the response omits this property.
188+
189+
`status`::
190+
(string)
191+
<<cluster-health,Health status>> of the data stream.
192+
+
193+
This health status is based on the state of the primary and replica shards of
194+
the stream's backing indices.
195+
+
196+
.Values for `status`
197+
[%collapsible%open]
198+
=====
199+
`GREEN`:::
200+
All shards are assigned.
201+
202+
`YELLOW`:::
203+
All primary shards are assigned, but one or more replica shards are
204+
unassigned.
205+
206+
`RED`:::
207+
One or more primary shards are unassigned, so some data is unavailable.
208+
=====
209+
210+
`template`::
211+
(string)
212+
Name of the index template used to create the data stream's backing indices.
213+
+
214+
The template's index pattern must match the name of this data stream. See
215+
<<create-index-template,create an index template>>.
216+
217+
`ilm_policy`::
218+
(string)
219+
Name of the current {ilm-init} lifecycle policy in the stream's matching index
220+
template. This lifecycle policy is set in the `index.lifecycle.name` setting.
221+
+
222+
If the template does not include a lifecycle policy, this property is not
223+
included in the response.
224+
+
225+
NOTE: A data stream's backing indices may be assigned different lifecycle
226+
policies. To retrieve the lifecycle policy for individual backing indices,
227+
use the <<indices-get-settings,get index settings API>>.
228+
229+
`next_generation_managed_by`::
230+
(string)
231+
Functionality in preview:[]. Indicates the system that will managed the next generation index
232+
(i.e. the next data stream write index).
233+
234+
`prefer_ilm`::
235+
(boolean)
236+
Functionality in preview:[]. Indicates if the index template used to create the data
237+
stream's backing indices is configured to prefer {ilm-cap} when both {ilm-cap} and
238+
<<data-stream-lifecycle, Data stream lifecycle>> are configured to manage this index.
239+
240+
`hidden`::
241+
(Boolean) If `true`, the data stream is <<multi-hidden,hidden>>.
242+
243+
`system`::
244+
(Boolean)
245+
If `true`, the data stream is created and managed by an Elastic stack component
246+
and cannot be modified through normal user interaction.
247+
248+
`allow_custom_routing`::
249+
(Boolean)
250+
If `true`, the data stream this data stream allows custom routing on write request.
251+
252+
`replicated`::
253+
(Boolean)
254+
If `true`, the data stream is created and managed by {ccr} and the local
255+
cluster can not write into this data stream or change its mappings.
256+
257+
`lifecycle`::
258+
(object)
259+
Functionality in preview:[]. Contains the configuration for the data stream lifecycle management of this data stream.
260+
+
261+
.Properties of `lifecycle`
262+
[%collapsible%open]
263+
=====
264+
`data_retention`::
265+
(string)
266+
If defined, every document added to this data stream will be stored at least for this time frame. Any time after this
267+
duration the document could be deleted. When empty, every document in this data stream will be stored indefinitely.
268+
269+
`rollover`::
270+
(object)
271+
The conditions which will trigger the rollover of a backing index as configured by the cluster setting
272+
`cluster.lifecycle.default.rollover`. This property is an implementation detail and it will only be retrieved when the query
273+
param `include_defaults` is set to `true`. The contents of this field are subject to change.
274+
=====
275+
276+
`rollover_on_write`::
277+
(Boolean)
278+
If `true`, the next write to this data stream will trigger a rollover first and the document will be
279+
indexed in the new backing index. If the rollover fails the indexing request will fail too.
280+
====
281+
282+
[[get-data-stream-api-example]]
283+
==== {api-examples-title}
284+
285+
[source,console]
286+
----
287+
GET _data_stream/my-data-stream*
288+
----
289+
290+
The API returns the following response:
291+
292+
[source,console-result]
293+
----
294+
{
295+
"data_streams": [
296+
{
297+
"name": "my-data-stream",
298+
"timestamp_field": {
299+
"name": "@timestamp"
300+
},
301+
"indices": [
302+
{
303+
"index_name": ".ds-my-data-stream-2099.03.07-000001",
304+
"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
305+
"prefer_ilm": true,
306+
"ilm_policy": "my-lifecycle-policy",
307+
"managed_by": "Index Lifecycle Management",
308+
"index_mode": "standard"
309+
},
310+
{
311+
"index_name": ".ds-my-data-stream-2099.03.08-000002",
312+
"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
313+
"prefer_ilm": true,
314+
"ilm_policy": "my-lifecycle-policy",
315+
"managed_by": "Index Lifecycle Management",
316+
"index_mode": "standard"
317+
}
318+
],
319+
"generation": 2,
320+
"_meta": {
321+
"my-meta-field": "foo"
322+
},
323+
"status": "GREEN",
324+
"index_mode": "standard",
325+
"next_generation_managed_by": "Index Lifecycle Management",
326+
"prefer_ilm": true,
327+
"template": "my-index-template",
328+
"ilm_policy": "my-lifecycle-policy",
329+
"hidden": false,
330+
"system": false,
331+
"allow_custom_routing": false,
332+
"replicated": false,
333+
"rollover_on_write": false
334+
},
335+
{
336+
"name": "my-data-stream-two",
337+
"timestamp_field": {
338+
"name": "@timestamp"
339+
},
340+
"indices": [
341+
{
342+
"index_name": ".ds-my-data-stream-two-2099.03.08-000001",
343+
"index_uuid": "3liBu2SYS5axasRt6fUIpA",
344+
"prefer_ilm": true,
345+
"ilm_policy": "my-lifecycle-policy",
346+
"managed_by": "Index Lifecycle Management",
347+
"index_mode": "standard"
348+
}
349+
],
350+
"generation": 1,
351+
"_meta": {
352+
"my-meta-field": "foo"
353+
},
354+
"status": "YELLOW",
355+
"index_mode": "standard",
356+
"next_generation_managed_by": "Index Lifecycle Management",
357+
"prefer_ilm": true,
358+
"template": "my-index-template",
359+
"ilm_policy": "my-lifecycle-policy",
360+
"hidden": false,
361+
"system": false,
362+
"allow_custom_routing": false,
363+
"replicated": false,
364+
"rollover_on_write": false
365+
}
366+
]
367+
}
368+
----
369+
// TESTRESPONSE[s/"index_name": ".ds-my-data-stream-2099.03.07-000001"/"index_name": $body.data_streams.0.indices.0.index_name/]
370+
// TESTRESPONSE[s/"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg"/"index_uuid": $body.data_streams.0.indices.0.index_uuid/]
371+
// TESTRESPONSE[s/"index_name": ".ds-my-data-stream-2099.03.08-000002"/"index_name": $body.data_streams.0.indices.1.index_name/]
372+
// TESTRESPONSE[s/"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw"/"index_uuid": $body.data_streams.0.indices.1.index_uuid/]
373+
// TESTRESPONSE[s/"index_name": ".ds-my-data-stream-two-2099.03.08-000001"/"index_name": $body.data_streams.1.indices.0.index_name/]
374+
// TESTRESPONSE[s/"index_uuid": "3liBu2SYS5axasRt6fUIpA"/"index_uuid": $body.data_streams.1.indices.0.index_uuid/]
375+
// TESTRESPONSE[s/"status": "GREEN"/"status": "YELLOW"/]
376+
// TESTRESPONSE[s/"replicated": false/"replicated": false,"failure_store":{"enabled": false, "indices": [], "rollover_on_write": true}/]

0 commit comments

Comments
 (0)