diff --git a/source/includes/abc-migration-intro.rst b/source/includes/abc-migration-intro.rst new file mode 100644 index 000000000..8a751e74d --- /dev/null +++ b/source/includes/abc-migration-intro.rst @@ -0,0 +1,4 @@ +Starting in {+c2c-beta-program-short+} 1.8, you can perform A->B->C migrations. +A->B->C migrations allow you to perform two consecutive migrations, where the +destination cluster of the first migration acts as the source cluster for the +second migration. diff --git a/source/includes/beta-feature.rst b/source/includes/beta-feature.rst index 57624ddd0..177d90ae5 100644 --- a/source/includes/beta-feature.rst +++ b/source/includes/beta-feature.rst @@ -1,4 +1,4 @@ .. important:: Cluster-to-Cluster Sync Beta Program - This feature is only available in ``mongosync-beta``. To learn more, see - :ref:`c2c-beta-program`. + This feature is only available in {+c2c-beta-program-short+}. To learn more, + see :ref:`c2c-beta-program`. diff --git a/source/includes/many-to-one-cluster.rst b/source/includes/many-to-one-cluster.rst new file mode 100644 index 000000000..7a1406592 --- /dev/null +++ b/source/includes/many-to-one-cluster.rst @@ -0,0 +1,4 @@ +Starting in {+c2c-beta-program-short+} 1.8, you can perform Many-to-One +migrations. Many-to-One migrations allow you to sync multiple source +clusters simultaneously with a destination cluster. For example, you can +consolidate data from many small clusters into a central cluster. diff --git a/source/includes/many-with-one-cluster.rst b/source/includes/many-with-one-cluster.rst deleted file mode 100644 index 0a7f2d88b..000000000 --- a/source/includes/many-with-one-cluster.rst +++ /dev/null @@ -1,3 +0,0 @@ -Starting in ``mongosync`` beta 1.8, you can sync multiple source -clusters simultaneously with a destination cluster. For example, you can -consolidate data from many small clusters into a central cluster. diff --git a/source/includes/migrationName-description.rst b/source/includes/migrationName-description.rst deleted file mode 100644 index cb518910e..000000000 --- a/source/includes/migrationName-description.rst +++ /dev/null @@ -1,4 +0,0 @@ -Starting in ``mongosync`` beta 1.8, you can set a migration name for a -sync operation. For example, you can set a migration name to identify -each sync operation from multiple source clusters into a destination -cluster. diff --git a/source/includes/migrationName.rst b/source/includes/migrationName.rst new file mode 100644 index 000000000..5d0f43071 --- /dev/null +++ b/source/includes/migrationName.rst @@ -0,0 +1,5 @@ +Starting in {+c2c-beta-program-short+} 1.8, you can specify a ``migrationName`` +to describe a sync operation. For example, you can set ``migrationName`` to +distinguish and identify different sync operations when working with +:ref:`Many-to-One ` or :ref:`A->B->C +` migrations. diff --git a/source/includes/opts/migrationName.rst b/source/includes/opts/migrationName.rst deleted file mode 100644 index 62040d382..000000000 --- a/source/includes/opts/migrationName.rst +++ /dev/null @@ -1,18 +0,0 @@ -.. reference/configuration.txt -.. reference/mongosync.txt - -Starting in ``mongosync`` beta 1.8, sets a migration name for a sync -operation. For example, you can set a migration name to identify each -sync operation from multiple source clusters into one destination -cluster. - -The ``migrationName`` string can contain up to 44 alphanumeric -and underscore characters. ``migrationName`` is appended to the string -``"mongosync_internal_"`` to set the migration metadata database name. - -If you set ``migrationName`` to -``"cluster_27000_to_cluster_35000_sync"``, the resulting ``mongosync`` -metadata database name is -``"mongosync_internal_cluster_27000_to_cluster_35000_sync"``. - -For a complete example, see :ref:`c2c-quickstart-many-with-one`. diff --git a/source/quickstart.txt b/source/quickstart.txt index af6a40336..514a854b6 100644 --- a/source/quickstart.txt +++ b/source/quickstart.txt @@ -307,35 +307,3 @@ Synchronization Notes - To estimate the size of ``oplog`` needed for initial synchronization, see :ref:`c2c-oplog-sizing`. - -.. _c2c-quickstart-many-with-one: - -Sync Multiple Source Clusters Simultaneously with a Destination Cluster -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. include:: /includes/many-with-one-cluster.rst - -For limitations, see :ref:`c2c-limitations_multiple_sync`. - -The following examples connect source clusters running on port ``27000`` -and ``27001`` with a destination cluster running on port ``35000``. The -commands also set the optional :option:`--migrationName` string to -describe the operations. - -.. code-block:: shell - - ./bin/mongosync \ - --cluster0 "mongodb://localhost:27000" \ - --cluster1 "mongodb://localhost:35000" \ - --migrationName "cluster_27000_to_cluster_35000_sync" - - ./bin/mongosync \ - --cluster0 "mongodb://localhost:27001" \ - --cluster1 "mongodb://localhost:35000" \ - --migrationName "cluster_27001_to_cluster_35000_sync" - -.. include:: /includes/many-with-one-cluster.rst - -To start the sync operation between the source clusters and the -destination cluster, see the earlier section -:ref:`c2c-quickstart-synchronize`. diff --git a/source/reference/beta-program.txt b/source/reference/beta-program.txt index ce03ec4f9..fcd01c95f 100644 --- a/source/reference/beta-program.txt +++ b/source/reference/beta-program.txt @@ -17,6 +17,8 @@ Each ``mongosync`` release has a corresponding {+c2c-beta-program-short+} build that includes its own set of experimental features. +.. _c2c-beta-program-disclaimer: + Get Started ----------- @@ -67,8 +69,8 @@ Beta Features * - Feature - Description - * - :ref:`c2c-beta-namespace-remapping` - - # DESCRIPTION TBD + * - :ref:`c2c-beta-abc-migration` + - .. include:: /includes/abc-migration-intro.rst * - :ref:`c2c-beta-document-filtering` - .. include:: /includes/document-filtering-intro.rst @@ -79,14 +81,22 @@ Beta Features * - :ref:`c2c-beta-destination-data-handling` - .. include:: /includes/destinationDataHandling-introduction.rst + * - :ref:`c2c-beta-migration-name` + - .. include:: /includes/migrationName.rst + + * - :ref:`c2c-beta-namespace-remapping` + - # DESCRIPTION TBD + * - :ref:`c2c-beta-many-to-one` - - .. include:: /includes/many-with-one-cluster.rst + - .. include:: /includes/many-to-one-cluster.rst .. toctree:: :titlesonly: + /reference/beta-program/abc-migration /reference/beta-program/namespace-remapping /reference/beta-program/document-filtering /reference/beta-program/orr /reference/beta-program/destinationDataHandling + /reference/beta-program/migration-name /reference/beta-program/many-to-one diff --git a/source/reference/beta-program/abc-migration.txt b/source/reference/beta-program/abc-migration.txt new file mode 100644 index 000000000..7db15b126 --- /dev/null +++ b/source/reference/beta-program/abc-migration.txt @@ -0,0 +1,89 @@ +.. _c2c-beta-abc-migration: + +================== +A->B->C Migrations +================== + +.. default-domain:: mongodb + +.. contents:: On this page + :local: + :backlinks: none + :depth: 2 + :class: singlecol + +.. include:: /includes/beta-feature.rst + +.. include:: /includes/abc-migration-intro.rst + +Syntax +------ + +To perform an A->B->C migration, run the following commands when starting +:ref:`mongosync `: + +.. code-block:: shell + + ./bin/mongosync \ + --cluster0 \ + --cluster1 \ + --migrationName + + ./bin/mongosync \ + --cluster0 \ + --cluster1 \ + --migrationName + +To start the sync operation between the source and destination clusters, see +:ref:`c2c-quickstart-synchronize`. + +Behavior +-------- + +The first migration (A->B) must finish :ref:`committing ` +before the second migration (B->C) starts committing. + +.. warning:: + + If the second migration starts committing before the first migration + finishes, you may experience data loss. + + Before you use any experimental {+c2c-beta-program-short+} features, + review the :ref:`{+c2c-full-beta-program+} Disclaimer + `. + +Example +------- + +The following example performs two consecutive migrations that: + +#. Connect a source cluster running on port ``27000`` with a destination + cluster running on port ``27001``. + +#. Use the destination cluster running on port ``27001`` as a source cluster + for the second migration. + +#. Connect the source cluster on port ``27001`` with a destination cluster + running on port ``27002``. + +The command also sets :ref:`--migrationName ` +to describe the operations and store migration metadata for each +sync. + +.. code-block:: shell + + ./bin/mongosync \ + --cluster0 "mongodb://localhost:27000" \ + --cluster1 "mongodb://localhost:27001" \ + --migrationName "cluster_27000_to_cluster_27001_sync" + + ./bin/mongosync \ + --cluster0 "mongodb://localhost:27001" \ + --cluster1 "mongodb://localhost:27002" \ + --migrationName "cluster_27001_to_cluster_27002_sync" + +Learn More +---------- + +- :ref:`c2c-beta-program` +- :ref:`c2c-beta-migration-name` diff --git a/source/reference/beta-program/document-filtering.txt b/source/reference/beta-program/document-filtering.txt index 4d85e5fe2..d4cf762ef 100644 --- a/source/reference/beta-program/document-filtering.txt +++ b/source/reference/beta-program/document-filtering.txt @@ -1,8 +1,8 @@ .. _c2c-beta-document-filtering: -================ -Filter Documents -================ +================== +Document Filtering +================== .. default-domain:: mongodb diff --git a/source/reference/beta-program/many-to-one.txt b/source/reference/beta-program/many-to-one.txt index c5bc91ef1..abf3a4338 100644 --- a/source/reference/beta-program/many-to-one.txt +++ b/source/reference/beta-program/many-to-one.txt @@ -1,8 +1,8 @@ .. _c2c-beta-many-to-one: -================================================================ -Synchronize Data Between Multiple Source Clusters Simultaneously -================================================================ +====================== +Many-to-One Migrations +====================== .. default-domain:: mongodb @@ -13,3 +13,80 @@ Synchronize Data Between Multiple Source Clusters Simultaneously :class: singlecol .. include:: /includes/beta-feature.rst + +.. include:: /includes/many-to-one-cluster.rst + +Syntax +------ + +To sync multiple source clusters with one destination cluster, run the following +commands when starting :ref:`mongosync `: + +.. code-block:: shell + + ./bin/mongosync \ + --cluster0 \ + --cluster1 \ + --migrationName + + ./bin/mongosync \ + --cluster0 \ + --cluster1 \ + --migrationName + +To start the sync operation between the source clusters and the +destination cluster, see :ref:`c2c-quickstart-synchronize`. + +Behavior +-------- + +A namespace is a ``database_name.collection_name`` combination. You can only +sync namespaces that don't conflict. + +For example, consider this scenario: + +- Two source clusters S1 and S2. +- A destination cluster D. +- Databases named ``inventory`` and ``sales`` on both S1 and S2. +- Collections named ``products``, ``orderLines``, ``orderStatus``, and + ``orders`` on both S1 and S2. +- You can sync both of these combinations: + + - ``inventory.products`` and ``sales.orderStatus`` on S1 with D. + - ``inventory.orderLines`` and ``sales.orders`` on S2 with D. + +- You cannot sync both of these combinations because they conflict: + + - ``inventory.products`` and ``inventory.orderLines`` on S1 with D. If D is + initially empty, you can sync S1 with D. ``inventory.products`` and + ``inventory.orderLines`` are copied from S1 to D. + - ``inventory.products`` and ``inventory.orderLines`` on S2 with D. You cannot + sync S2 with D because ``inventory.products`` and ``inventory.orderLines`` + conflict with the namespaces already on D from the scenario in the previous + point. + +Example +------- + +The following example connects source clusters running on port ``27000`` +and ``27001`` with a destination cluster running on port ``35000``. The command +also sets the :ref:`--migrationName ` option to +describe the operations and store migration metadata for each sync. + +.. code-block:: shell + + ./bin/mongosync \ + --cluster0 "mongodb://localhost:27000" \ + --cluster1 "mongodb://localhost:35000" \ + --migrationName "cluster_27000_to_cluster_35000_sync" + + ./bin/mongosync \ + --cluster0 "mongodb://localhost:27001" \ + --cluster1 "mongodb://localhost:35000" \ + --migrationName "cluster_27001_to_cluster_35000_sync" + +Learn More +---------- + +- :ref:`c2c-beta-program` +- :ref:`c2c-beta-migration-name` diff --git a/source/reference/beta-program/migration-name.txt b/source/reference/beta-program/migration-name.txt new file mode 100644 index 000000000..d6354fd87 --- /dev/null +++ b/source/reference/beta-program/migration-name.txt @@ -0,0 +1,65 @@ +.. _c2c-beta-migration-name: + +============= +migrationName +============= + +.. default-domain:: mongodb + +.. contents:: On this page + :local: + :backlinks: none + :depth: 2 + :class: singlecol + +.. include:: /includes/beta-feature.rst + +.. include:: /includes/migrationName.rst + +Syntax +------ + +To set ``migrationName`` from the command line, use the ``--migrationName`` +setting: + +.. code-block:: shell + + ./bin/mongosync \ + --cluster0 \ + --cluster1 \ + --migrationName + +To set ``migrationName`` in a :ref:`configuration file `, +include the ``migrationName`` option: + +.. code-block:: yaml + + cluster0: + cluster1: + migrationName: + +Behavior +-------- + +The ``migrationName`` string can contain up to 44 alphanumeric and underscore +characters. ``migrationName`` is appended to the string +``"mongosync_internal_"`` to set the migration metadata database name. + +For example, if you set ``migrationName`` to +``"cluster_27000_to_cluster_35000_sync"``, the resulting ``mongosync`` metadata +database name is ``"mongosync_internal_cluster_27000_to_cluster_35000_sync"``. + +.. _c2c-beta-multiple-migration-examples: + +Examples +-------- + +To view examples that use ``migrationName``, see: + +- :ref:`c2c-beta-abc-migration` +- :ref:`c2c-beta-many-to-one` + +Learn More +---------- + +- :ref:`c2c-beta-program` diff --git a/source/reference/configuration.txt b/source/reference/configuration.txt index a624e5b7c..d3b8dd988 100644 --- a/source/reference/configuration.txt +++ b/source/reference/configuration.txt @@ -111,15 +111,6 @@ Options To set the ``logPath`` setting from the command line, see the :option:`--logPath` option. -.. setting:: migrationName - - *Type*: string - - .. include:: /includes/opts/migrationName.rst - - To set the ``migrationName`` setting from the command line, see the - :option:`--migrationName` option. - .. setting:: port *Type*: integer diff --git a/source/reference/limitations.txt b/source/reference/limitations.txt index 73ee8a5a9..4be1bc201 100644 --- a/source/reference/limitations.txt +++ b/source/reference/limitations.txt @@ -55,41 +55,6 @@ General Limitations - ``mongosync`` must read from the source cluster using the :readmode:`primary` read preference. -.. _c2c-limitations_multiple_sync: - -Multiple Source Sync Limitations --------------------------------- - -.. include:: /includes/many-with-one-cluster.rst - -A namespace is a ``database_name.collection_name`` combination. -You can only sync namespaces that don't conflict. - -For example, consider this scenario: - -- Two source clusters S1 and S2. -- A destination cluster D. -- Databases named ``inventory`` and ``sales`` on both S1 and S2. -- Collections named ``products``, ``orderLines``, ``orderStatus``, and - ``orders`` on both S1 and S2. -- You can sync both of these combinations: - - - ``inventory.products`` and ``sales.orderStatus`` on S1 with D. - - ``inventory.orderLines`` and ``sales.orders`` on S2 with D. - -- You cannot sync both of these combinations because they conflict: - - - ``inventory.products`` and ``inventory.orderLines`` on S1 with D. If - D is initially empty, you can sync S1 with D. ``inventory.products`` - and ``inventory.orderLines`` are copied from S1 to D. - - ``inventory.products`` and ``inventory.orderLines`` on S2 with D. - You cannot sync S2 with D because ``inventory.products`` and - ``inventory.orderLines`` conflict with the namespaces already on D - from the scenario in the previous point. - -Additionally, for views, you must ensure source view names don't -conflict with existing destination view names. - MongoDB Community Edition ------------------------- diff --git a/source/reference/mongosync.txt b/source/reference/mongosync.txt index 0cdb35a6e..cb263e4db 100644 --- a/source/reference/mongosync.txt +++ b/source/reference/mongosync.txt @@ -119,13 +119,6 @@ Global Options .. note:: .. include:: /includes/fact-log-rotation-usr1-signal - -.. option:: --migrationName - - .. include:: /includes/opts/migrationName.rst - - To set the ``--migrationName`` option from a configuration file, - see the :setting:`migrationName` setting. .. option:: --port diff --git a/source/release-notes/1.8.txt b/source/release-notes/1.8.txt index cf68f3c71..e85229f8f 100644 --- a/source/release-notes/1.8.txt +++ b/source/release-notes/1.8.txt @@ -28,6 +28,13 @@ Release Notes for mongosync 1.8 To learn more, see :ref:`c2c-beta-program`. +A->B->C Migrations +`````````````````` + +.. include:: /includes/abc-migration-intro.rst + +To learn more, see :ref:`c2c-beta-abc-migration`. + Document Filtering in mongosync ``````````````````````````````` @@ -36,26 +43,25 @@ Document Filtering in mongosync To learn more, see :ref:`c2c-beta-document-filtering`. Handle Destination Data with destinationDataHandling Option -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +``````````````````````````````````````````````````````````` .. include:: /includes/destinationDataHandling-introduction.rst For details, see :ref:`c2c-beta-destination-data-handling`. -Synchronize Data Between Multiple Source Clusters Simultaneously -```````````````````````````````````````````````````````````````` - -.. include:: /includes/many-with-one-cluster.rst - Set a Migration Name ```````````````````` -.. include:: /includes/migrationName-description.rst +.. include:: /includes/migrationName.rst + +To learn more, see :ref:`c2c-beta-migration-name`. + +Many-to-One Migrations +`````````````````````` -To set a migration name, see: +.. include:: /includes/many-to-one-cluster.rst -- :setting:`migrationName` configuration file setting -- :option:`--migrationName` command line option +To learn more, see :ref:`c2c-beta-many-to-one`. Other Notes ~~~~~~~~~~~