forked from mongodb/docs-cluster-to-cluster-sync
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathquickstart.txt
316 lines (228 loc) · 11.2 KB
/
quickstart.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
.. _c2c-quickstart:
==================================
{+c2c-product-name+} Quickstart
==================================
.. default-domain:: mongodb
.. facet::
:name: genre
:values: tutorial
.. contents:: On this page
:local:
:backlinks: none
:depth: 2
:class: singlecol
Overview
--------
{+c2c-full-product-name+} connects MongoDB clusters and provides a way
to synchronize data between them. The tool that makes the connection is
:ref:`c2c-mongosync`. This page provides a short introduction to help
you get started with {+c2c-product-name+}. For more details, please see
the rest of the {+c2c-product-name+} documentation.
``mongosync`` syncs data between two clusters.
- Each cluster can be a replica set or a sharded cluster. If either
is a sharded cluster, consult the :ref:`sharded cluster limitations
<c2c-sharded-limitations>` for ``mongosync``.
- Sharded clusters are not required to have the same number of shards.
- The destination cluster must be the same version or up to two versions ahead
of the source cluster. The patch version is not important, so long as they
meets the minimum patch :ref:`version requirements
<c2c-server-version-compatibility>`.
Follow the instructions below to set up {+c2c-product-name+}, connect
your clusters, and synchronize your data.
Setup
-----
.. procedure::
:style: normal
.. step:: Define a source and a destination cluster
If you already have a MongoDB cluster, either self-managed or
hosted in :atlas:`MongoDB Atlas </getting-started?jmp=docs>`,
use that cluster as the source cluster. If you don't have a
cluster to work with, you will need to create one.
This Quickstart works when the destination cluster and the source
cluster are both replica sets. To sync from a replica set to a
sharded cluster, or between sharded clusters, see:
:ref:`c2c-sharded-clusters`.
.. seealso::
You can migrate between clusters on the same or different
MongoDB versions. For more information on MongoDB server
version compatibility or cross-version migrations, see
:ref:`c2c-server-version-compatibility`.
The number of nodes in the destination replica set does not have
to equal the number of nodes in the source replica set.
If your clusters are self-managed, they must be MongoDB
Enterprise clusters. {+c2c-product-name+} is only supported on
MongoDB Community Edition in a limited number of cases. For more
information on using {+c2c-product-name+} with MongoDB Community
Edition, contact a MongoDB sales representative.
Creating a cluster is beyond the scope of this guide. If you need
help, refer to the documentation to create an :ref:`Atlas cluster
<create-new-cluster>` or to create a :ref:`self-managed cluster
<server-replica-set-deploy>`.
.. step:: Define administrative users
If either cluster is hosted in Atlas, or if either of them
requires authentication, you must create a database user that has
permissions in both clusters.
Source Cluster Authentication Requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The source user must have the following roles:
- :authrole:`readAnyDatabase` role
- :authrole:`clusterMonitor` role
- :authrole:`backup` role
In addition, the source user must be able to:
- Run the :dbcommand:`getParameter` command
If the source cluster is hosted in Atlas, the user must have the
:ref:`Atlas admin <atlas-user-privileges>` role. The user must
also be able to read the :ref:`change stream <changeStreams>` for
the cluster.
Destination Cluster Authentication Requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If the destination cluster is hosted in Atlas, the user must have
the :ref:`Atlas admin <atlas-user-privileges>` role.
Cluster Authentication Notes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- To add an Atlas user, see: :ref:`mongodb-users`.
- To add a user to a self-managed cluster, see: :ref:`create-users`.
- To verify user permissions, run :method:`db.getUser()`.
.. step:: Download and install ``mongosync``
:ref:`c2c-mongosync` is the tool that connects the source and
destination clusters. You can host ``mongosync`` on its own
hardware, ``mongosync`` does not have to run on the hardware that
hosts your MongodDB clusters.
To install ``mongosync``:
#. Locate a host server for the ``mongosync`` executable that has
network connectivity to your source and destination clusters.
#. Go to the {+mdb-download-center+}.
#. Download the ``mongosync`` package for your host system.
#. Unpack the ``mongosync`` package. The ``mongosync`` executable
is in the ``bin`` directory.
For operating system specific installation instructions, see
:ref:`c2c-install`.
Connect the Clusters
--------------------
.. procedure::
:style: normal
.. step:: Format your connection strings
A connection string contains the network and authentication
details that ``mongosync`` needs to connect to the source and
destination clusters.
Determine the hostname or IP address and port for your source and
destination clusters. You will use this information and the user
authentication details to construct the connection strings.
The standard URI connection scheme has the form:
.. code-block:: shell
mongodb://[username:password@]host1[:port1][,...hostN[:portN]][/[defaultauthdb][?options]]
Your connections strings will resemble:
.. code-block:: shell
cluster0:
mongodb://clusterAdmin:[email protected]:20020,clusterOne02.fancyCorp.com:20020,clusterOne03.fancyCorp.com:20020
cluster1:
mongodb://clusterAdmin:[email protected]:20020,clusterTwo02.fancyCorp.com:20020,clusterTwo03.fancyCorp.com:20020
For more details, see :ref:`c2c-connecting`.
.. step:: Initialize mongosync
:program:`mongosync` must create an initial connection to the source and
destination clusters before it can start to sync data. To create the initial
connection, issue the following command with your connection
strings on a single line (the command is
reformatted here for clarity):
.. code-block:: shell
./bin/mongosync \
--logPath /var/log/mongosync \
--cluster0 "mongodb://clusterAdmin:[email protected]:20020,clusterOne02.fancyCorp.com:20020,clusterOne03.fancyCorp.com:20020" \
--cluster1 "mongodb://clusterAdmin:[email protected]:20020,clusterTwo02.fancyCorp.com:20020,clusterTwo03.fancyCorp.com:20020"
Initialization Notes
~~~~~~~~~~~~~~~~~~~~
- When ``mongosync`` first connects to the source and destination
clusters it is in the :ref:`IDLE <c2c-states-descriptions>` state.
- ``mongosync`` does not synchronize data until it receives the
:ref:`start <c2c-api-start>` command.
- Designate the source and destination clusters with the :ref:`start
<c2c-api-start>` command. "cluster0" and "cluster1" are just labels,
either cluster can be ``cluster0`` or ``cluster1``.
Synchronize Data Between Clusters
---------------------------------
The :ref:`start <c2c-api-start>` endpoint initiates data synchronization.
To start syncing, use ``curl`` or a similar program to issue the
:ref:`start <c2c-api-start>` request:
.. literalinclude:: /includes/api/requests/start.sh
:language: shell
If the :ref:`start <c2c-api-start>` request is successful, ``mongosync``
returns ``{ "success": true }`` and starts to synchronize existing data on
the source cluster with the destination cluster. At this point,
``mongosync`` enters the ``RUNNING`` state and applies subsequent source
cluster writes to the destination cluster.
To check the status of the sync, call the :ref:`progress
<c2c-api-progress>` endpoint:
.. literalinclude:: /includes/api/requests/progress.sh
:language: shell
If the ``progress`` response includes the
field ``canCommit: true``, the clusters are in sync and the destination
cluster continuously replicates data from the source cluster.
The command interface for ``mongosync`` is an HTTP server that publishes
an HTTP API. To control ``mongosync``, use the API endpoints. The API
documentation provides details on using the following endpoints:
.. list-table::
:header-rows: 1
* - Endpoint
- Description
* - :ref:`c2c-api-start`
- Starts the synchronization between a source and destination
cluster.
* - :ref:`c2c-api-progress`
- Returns the status of the synchronization process.
* - :ref:`c2c-api-pause`
- Pauses the current synchronization operation.
* - :ref:`c2c-api-resume`
- Resumes a paused synchronization session based on data stored on
the destination cluster.
* - :ref:`c2c-api-commit`
- Commits the synchronization operation to the destination
cluster.
* - :ref:`c2c-api-reverse`
- Reverses the direction of a committed sync operation.
Finalize Cutover Process
------------------------
You can finalize a migration and transfer your application
workload from the source to the destination cluster using the
``mongosync`` cutover process.
For more information, see :ref:`c2c-cutover-process`.
One-Time Sync
-------------
After initializing data synchronization, call the :ref:`progress
<c2c-api-progress>` endpoint to see the status of the synchronization
process:
.. literalinclude:: /includes/api/requests/progress.sh
:language: shell
For a one time sync, verify that the ``progress`` response includes the
following field values:
- ``state: "RUNNING"``
- ``canCommit: true``
- ``lagTimeSeconds`` is near ``0`` (*Recommended, but not required*)
Then, call the :ref:`commit <c2c-api-commit>` endpoint to commit the
synchronization operation to the destination cluster and stop continuous
replication:
.. literalinclude:: /includes/api/requests/commit.sh
:language: shell
If the ``commit`` request is successful, ``mongosync`` returns
``{ "success": true }`` and enters the ``COMMITTING`` state. After the sync
is complete, ``mongosync`` enters the ``COMMITTED`` state and the
clusters are no longer in continuous sync.
Data Verification
-----------------
.. include:: /includes/fact-verification
Synchronization Notes
---------------------
- The default port for the HTTP API is ``27182``. Use the ``--port``
option with ``mongosync`` to :ref:`configure another port
<c2c-mongosync>`
- ``mongosync`` can swap the source and destination clusters to enable
:ref:`reverse synchronization <c2c-api-reverse>`.
For more information, see the :ref:`c2c-api-reverse` endpoint.
- .. include:: /includes/fact-permissions-body.rst
To determine the correct the user permissions for your use case, see
:ref:`c2c-permissions-and-roles`.
- You may need to increase the file descriptor ``ulimits`` on the host
that is running ``mongosync``. This applies to any UNIX-like system,
but macOS in particular has low defaults. See :ref:`UNIX ulimit
settings <system-resource-utilization>`.
- To estimate the size of ``oplog`` needed for initial synchronization,
see :ref:`c2c-oplog-sizing`.