Skip to content

Commit 92fef40

Browse files
authored
Introduce CCR getting started guide (elastic#35434)
This commit introduces a basic getting started guide for cross-cluster replication to the docs. Co-authored-by: "lcawl <[email protected]>"
1 parent 76cefb0 commit 92fef40

File tree

1 file changed

+324
-2
lines changed

1 file changed

+324
-2
lines changed
Lines changed: 324 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,329 @@
11
[role="xpack"]
22
[testenv="platinum"]
33
[[ccr-getting-started]]
4-
== Getting Started
4+
== Getting Started with {ccr}
55

66
beta[]
7-
This is the getting started section of the {ccr} docs.
7+
8+
This getting-started guide for {ccr} shows you how to:
9+
10+
* <<ccr-getting-started-remote-cluster,Connect a local cluster to a remote
11+
cluster>>
12+
* <<ccr-getting-started-leader-index,Create a leader index>> in a remote cluster
13+
* <<ccr-getting-started-follower-index,Create a follower index>> that replicates
14+
a leader index
15+
* <<ccr-getting-started-auto-follow,Automatically create follower indices>>
16+
17+
[float]
18+
[[ccr-getting-started-before-you-begin]]
19+
=== Before you begin
20+
. {stack-gs}/get-started-elastic-stack.html#install-elasticsearch[Install {es}]
21+
on your local and remote clusters.
22+
23+
. Obtain a license that includes the {ccr} features. See
24+
https://www.elastic.co/subscriptions[subscriptions] and
25+
<<license-management>>.
26+
27+
. If the Elastic {security-features} are enabled in your local and remote
28+
clusters, you need a user that has appropriate authority to perform the steps
29+
in this tutorial.
30+
+
31+
--
32+
[[ccr-getting-started-security]]
33+
The {ccr} features use cluster privileges and built-in roles to make it easier
34+
to control which users have authority to manage {ccr}.
35+
36+
By default, you can perform all of the steps in this tutorial by
37+
using the built-in `elastic` user. However, a password must be set for this user
38+
before the user can do anything. For information about how to set that password,
39+
see <<security-getting-started>>.
40+
41+
If you are performing these steps in a production environment, take extra care
42+
because the `elastic` user has the `superuser` role and you could inadvertently
43+
make significant changes.
44+
45+
Alternatively, you can assign the appropriate privileges to a user ID of your
46+
choice. On the remote cluster that contains the leader index, a user will need
47+
the `read_ccr` cluster privilege and `monitor` and `read` privileges on the
48+
leader index.
49+
50+
[source,yml]
51+
--------------------------------------------------
52+
ccr_user:
53+
cluster:
54+
- read_ccr
55+
indices:
56+
- names: [ 'leader-index' ]
57+
privileges:
58+
- monitor
59+
- read
60+
--------------------------------------------------
61+
62+
On the local cluster that contains the follower index, the same user will need
63+
the `manage_ccr` cluster privilege and `monitor`, `read`, `write` and
64+
`manage_follow_index` privileges on the follower index.
65+
66+
[source,yml]
67+
--------------------------------------------------
68+
ccr_user:
69+
cluster:
70+
- manage_ccr
71+
indices:
72+
- names: [ 'follower-index' ]
73+
privileges:
74+
- monitor
75+
- read
76+
- write
77+
- manage_follow_index
78+
--------------------------------------------------
79+
80+
If you are managing
81+
<<ccr-getting-started-remote-cluster,connecting to the remote cluster>> via the
82+
cluster update settings API, you will also need a user with the `all` cluster
83+
privilege.
84+
--
85+
86+
[float]
87+
[[ccr-getting-started-remote-cluster]]
88+
=== Connecting to a remote cluster
89+
90+
The {ccr} features require that you
91+
{ref}/modules-remote-clusters.html[connect your local cluster to a remote
92+
cluster]. In this tutorial, we will connect our local cluster to a remote
93+
cluster with the cluster alias `leader`.
94+
95+
[source,js]
96+
--------------------------------------------------
97+
PUT /_cluster/settings
98+
{
99+
"persistent" : {
100+
"cluster" : {
101+
"remote" : {
102+
"leader" : {
103+
"seeds" : [
104+
"127.0.0.1:9300" <1>
105+
]
106+
}
107+
}
108+
}
109+
}
110+
}
111+
--------------------------------------------------
112+
// CONSOLE
113+
// TEST[setup:host]
114+
// TEST[s/127.0.0.1:9300/\${transport_host}/]
115+
<1> Specifies the hostname and transport port of a seed node in the remote
116+
cluster.
117+
118+
You can verify that the local cluster is successfully connected to the remote
119+
cluster.
120+
121+
[source,js]
122+
--------------------------------------------------
123+
GET /_remote/info
124+
--------------------------------------------------
125+
// CONSOLE
126+
// TEST[continued]
127+
128+
The API will respond by showing that the local cluster is connected to the
129+
remote cluster.
130+
131+
[source,js]
132+
--------------------------------------------------
133+
{
134+
"leader" : {
135+
"seeds" : [
136+
"127.0.0.1:9300"
137+
],
138+
"connected" : true, <1>
139+
"num_nodes_connected" : 1, <2>
140+
"max_connections_per_cluster" : 3,
141+
"initial_connect_timeout" : "30s",
142+
"skip_unavailable" : false
143+
}
144+
}
145+
--------------------------------------------------
146+
// TESTRESPONSE
147+
// TEST[s/127.0.0.1:9300/$body.leader.seeds.0/]
148+
// TEST[s/"connected" : true/"connected" : $body.leader.connected/]
149+
// TEST[s/"num_nodes_connected" : 1/"num_nodes_connected" : $body.leader.num_nodes_connected/]
150+
<1> This shows the local cluster is connected to the remote cluster with cluster
151+
alias `leader`
152+
<2> This shows the number of nodes in the remote cluster the local cluster is
153+
connected to.
154+
155+
[float]
156+
[[ccr-getting-started-leader-index]]
157+
=== Creating a leader index
158+
159+
Leader indices require special index settings to ensure that the operations that
160+
need to be replicated are available when the
161+
follower requests them from the leader. These settings are used to enable soft
162+
deletes on the leader index and to control how many soft deletes are retained. A
163+
_soft delete_ occurs whenever a document is deleted or updated. Soft deletes can
164+
be enabled only on new indices created on or after {es} 6.5.0.
165+
166+
In the following example, we will create a leader index in the remote cluster:
167+
168+
[source,js]
169+
--------------------------------------------------
170+
PUT /server-metrics
171+
{
172+
"settings" : {
173+
"index" : {
174+
"number_of_shards" : 1,
175+
"number_of_replicas" : 0,
176+
"soft_deletes" : {
177+
"enabled" : true, <1>
178+
"retention" : {
179+
"operations" : 1024 <2>
180+
}
181+
}
182+
}
183+
},
184+
"mappings" : {
185+
"metric" : {
186+
"properties" : {
187+
"@timestamp" : {
188+
"type" : "date"
189+
},
190+
"accept" : {
191+
"type" : "long"
192+
},
193+
"deny" : {
194+
"type" : "long"
195+
},
196+
"host" : {
197+
"type" : "keyword"
198+
},
199+
"response" : {
200+
"type" : "float"
201+
},
202+
"service" : {
203+
"type" : "keyword"
204+
},
205+
"total" : {
206+
"type" : "long"
207+
}
208+
}
209+
}
210+
}
211+
}
212+
--------------------------------------------------
213+
// CONSOLE
214+
// TEST[continued]
215+
<1> Enables soft deletes on the leader index.
216+
<2> Sets that up to 1024 soft deletes will be retained.
217+
218+
[float]
219+
[[ccr-getting-started-follower-index]]
220+
=== Creating a follower index
221+
222+
Follower indices are created with the {ref}/ccr-put-follow.html[create follower
223+
API]. When you create a follower index, you must reference the
224+
<<ccr-getting-started-remote-cluster,remote cluster>> and the
225+
<<ccr-getting-started-leader-index,leader index>> that you created in the remote
226+
cluster.
227+
228+
[source,js]
229+
--------------------------------------------------
230+
PUT /server-metrics-copy/_ccr/follow
231+
{
232+
"remote_cluster" : "leader",
233+
"leader_index" : "server-metrics"
234+
}
235+
--------------------------------------------------
236+
// CONSOLE
237+
// TEST[continued]
238+
239+
//////////////////////////
240+
241+
[source,js]
242+
--------------------------------------------------
243+
{
244+
"follow_index_created" : true,
245+
"follow_index_shards_acked" : true,
246+
"index_following_started" : true
247+
}
248+
--------------------------------------------------
249+
// TESTRESPONSE
250+
251+
//////////////////////////
252+
253+
Now when you index documents into your leader index, you will see these
254+
documents replicated in the follower index. You can
255+
inspect the status of replication using the
256+
{ref}/ccr-get-follow-stats[get follower stats API].
257+
258+
//////////////////////////
259+
260+
[source,js]
261+
--------------------------------------------------
262+
POST /server-metrics-copy/_ccr/pause_follow
263+
264+
POST /server-metrics-copy/_close
265+
266+
POST /server-metrics-copy/_ccr/unfollow
267+
--------------------------------------------------
268+
// CONSOLE
269+
// TEST[continued]
270+
271+
//////////////////////////
272+
273+
[float]
274+
[[ccr-getting-started-auto-follow]]
275+
=== Automatically create follower indices
276+
277+
The auto-follow feature in {ccr} helps for time series use cases where you want
278+
to follow new indices that are periodically created in the remote cluster
279+
(such as daily Beats indices). Auto-following is configured using the
280+
{ref}/ccr-put-auto-follow-pattern.html[create auto-follow pattern API]. With an
281+
auto-follow pattern, you reference the
282+
<<ccr-getting-started-remote-cluster,remote cluster>> that you connected your
283+
local cluster to. You must also specify a collection of patterns that match the
284+
indices you want to automatically follow.
285+
286+
For example:
287+
288+
[source,js]
289+
--------------------------------------------------
290+
PUT /_ccr/auto_follow/beats
291+
{
292+
"remote_cluster" : "leader",
293+
"leader_index_patterns" :
294+
[
295+
"metricbeat-*", <1>
296+
"packetbeat-*" <2>
297+
],
298+
"follow_index_pattern" : "{{leader_index}}-copy" <3>
299+
}
300+
--------------------------------------------------
301+
// CONSOLE
302+
// TEST[continued]
303+
<1> Automatically follow new {metricbeat} indices.
304+
<2> Automatically follow new {packetbeat} indices.
305+
<3> The name of the follower index is derived from the name of the leader index
306+
by adding the suffix `-copy` to the name of the leader index.
307+
308+
//////////////////////////
309+
310+
[source,js]
311+
--------------------------------------------------
312+
{
313+
"acknowledged" : true
314+
}
315+
--------------------------------------------------
316+
// TESTRESPONSE
317+
318+
//////////////////////////
319+
320+
//////////////////////////
321+
322+
[source,js]
323+
--------------------------------------------------
324+
DELETE /_ccr/auto_follow/beats
325+
--------------------------------------------------
326+
// CONSOLE
327+
// TEST[continued]
328+
329+
//////////////////////////

0 commit comments

Comments
 (0)