Add documentation for remote cluster proxy mode

This is related to elastic#49067.

Author: Tim-Brooks

docs/reference/cluster/remote-info.asciidoc

@@ -17,29 +17,47 @@ Returns configured remote cluster information.
 ==== {api-description-title}
 
 The cluster remote info API allows you to retrieve all of the configured
-remote cluster information. It returns connection and endpoint information keyed 
+remote cluster information. It returns connection and endpoint information keyed
 by the configured remote cluster alias.
 
 
 [[cluster-remote-info-api-response-body]]
 ==== {api-response-body-title}
 
-`seeds`::
-	The configured initial seed transport addresses of the remote cluster.
+`mode`::
+    The configured connection mode for the remote cluster. Will be `sniff` or
+    `proxy`.
 
 `connected`::
 	True if there is at least one connection to the remote cluster.
 
-`num_nodes_connected`::
-    The number of connected nodes in the remote cluster.
-
-`max_connections_per_cluster`::
-	The maximum number of connections maintained for the remote cluster.
-
 `initial_connect_timeout`::
 	The initial connect timeout for remote cluster connections.
 
 [[skip-unavailable]]
 `skip_unavailable`::
     Whether the remote cluster is skipped in case it is searched through
     a {ccs} request but none of its nodes are available.
+
+`seeds`::
+	The configured initial seed transport addresses of the remote cluster when
+	sniff mode is configured.
+
+`num_nodes_connected`::
+    The number of connected nodes in the remote cluster when sniff mode is
+    configured.
+
+`max_connections_per_cluster`::
+	The maximum number of connections maintained for the remote cluster when
+	sniff mode is configured.
+
+`address`::
+    The configured address for remote connections when proxy mode is configured.
+
+`num_sockets_connected`::
+    The number of open socket connections to the remote cluster when proxy mode
+    is configured.
+
+`max_socket_connections`::
+	The maximum number of socket connections to the remote cluster when proxy
+	mode is configured.

docs/reference/modules/remote-clusters.asciidoc

@@ -15,21 +15,32 @@ endif::[]
 
 // tag::how-remote-clusters-work[]
 Remote cluster connections work by configuring a remote cluster and connecting
-only to a limited number of nodes in that remote cluster. Each remote cluster
-is referenced by a name and a list of seed nodes. When a remote cluster is
-registered, its cluster state is retrieved from one of the seed nodes and up
-to three _gateway nodes_ are selected to be connected to as part of remote
-cluster requests.
-// end::how-remote-clusters-work[]
+only to a limited number of nodes in that remote cluster. There are two
+potential modes for remote cluster connections: sniff mode and proxy mode.
 
 All the communication required between different clusters
 goes through the <<modules-transport,transport layer>>. Remote cluster
 connections consist of uni-directional connections from the coordinating
-node to the selected remote _gateway nodes_ only.
+node to the remote remote connections.
+// end::how-remote-clusters-work[]
+
+[float]
+[[sniff-mode]]
+=== Sniff mode
+
+// tag::how-sniff-mode-works[]
+A cluster in sniff mode is created by a name and a list of seed nodes. When a
+remote cluster is registered, its cluster state is retrieved from one of the
+seed nodes and up to three _gateway nodes_ are selected to be connected to as
+part of remote cluster requests. This mode requires that the _gateway node_
+publish addresses are exposed to the local cluster.
+
+The sniff mode is the default connection mode.
+// end::how-remote-clusters-work[]
 
 [float]
 [[gateway-nodes-selection]]
-=== Gateway nodes selection
+==== Gateway nodes selection
 
 The _gateway nodes_ selection depends on the following criteria:
 
@@ -62,9 +73,26 @@ communicate with 6.7. The matrix below summarizes compatibility as described abo
 (see <<remote-cluster-settings>>), though such tagged nodes still have
 to satisfy the two above requirements.
 
+[float]
+[[proxy-mode]]
+=== Proxy mode
+
+// tag::how-proxy-mode-works[]
+A cluster in proxy mode is created by a name and a single proxy address. When
+a remote cluster is registered, a configurable number of socket connections are
+opened to the proxy address. The proxy is required to route those connections to
+the remote cluster. Proxy mode does not require that the remote cluster nodes
+publish addresses are accessible.
+
+The proxy mode is not the default connection mode and must be configured. Similar
+to the sniff <<remote-cluster-compatibility-matrix,gateway nodes>>, the remote
+connections are subject to the same version compatibility rules as
+<<rolling-upgrades>>.
+// end::how-remote-clusters-work[]
+
 [float]
 [[configuring-remote-clusters]]
-=== Configuring remote clusters
+==== Configuring remote clusters
 
 You can configure remote clusters globally by using
 <<cluster-update-settings,cluster settings>>, which you can update dynamically.
@@ -83,23 +111,32 @@ cluster:
         cluster_one: <1>
             seeds: 127.0.0.1:9300 <2>
             transport.ping_schedule: 30s <3>
-        cluster_two: 
-            seeds: 127.0.0.1:9301
-            transport.compress: true <4>
-            skip_unavailable: true <5>
+        cluster_two: <1>
+            mode: sniff <4>
+            seeds: 127.0.0.1:9301 <2>
+            transport.compress: true <5>
+            skip_unavailable: true <6>
+        cluster_three: <1>
+            mode: proxy <4>
+            proxy_address: 127.0.0.1:9302 <7>
 
 --------------------------------
-<1> `cluster_one` and `cluster_two` are arbitrary _cluster aliases_ representing
-the connection to each cluster. These names are subsequently used to distinguish
-between local and remote indices.
+<1> `cluster_one`, `cluster_two`, and `cluster_three` are arbitrary _cluster aliases_
+representing the connection to each cluster. These names are subsequently used to
+distinguish between local and remote indices.
 <2> The hostname and <<modules-transport,transport>> port (default: 9300) of a
 seed node in the remote cluster.
 <3> A keep-alive ping is configured for `cluster_one`.
-<4> Compression is explicitly enabled for requests to `cluster_two`.
-<5> Disconnected remote clusters are optional for `cluster_two`.
+<4> The configured connection mode. By default it is `sniff`, so the mode is implicit
+for `cluster_one`. However, it can be explicitly configured as demonstrated by
+`cluster_two` and must be explicitly configured for `proxy` mode as demonstrated by
+`cluster_three`.
+<5> Compression is explicitly enabled for requests to `cluster_two`.
+<6> Disconnected remote clusters are optional for `cluster_two`.
+<7> The address for the proxy endpoint used to connect to `cluster_three`.
 
 For more information about the optional transport settings, see
-<<modules-transport>>. 
+<<modules-transport>>.
 
 
 If you use <<cluster-update-settings,cluster settings>>, the remote clusters
@@ -119,16 +156,16 @@ PUT _cluster/settings
           "transport.ping_schedule": "30s"
         },
         "cluster_two": {
+          "mode": "sniff",
           "seeds": [
             "127.0.0.1:9301"
           ],
           "transport.compress": true,
           "skip_unavailable": true
         },
         "cluster_three": {
-          "seeds": [
-            "127.0.0.1:9302"
-          ]
+          "mode": "proxy",
+          "proxy_address": "127.0.0.1:9302"
         }
       }
     }
@@ -139,7 +176,8 @@ PUT _cluster/settings
 // TEST[s/127.0.0.1:9300/\${transport_host}/]
 
 You can dynamically update the compression and ping schedule settings. However,
-you must re-include seeds in the settings update request. For example:
+you must re-include seeds or proxy_address in the settings update request.
+For example:
 
 [source,console]
 --------------------------------
@@ -155,10 +193,16 @@ PUT _cluster/settings
           "transport.ping_schedule": "60s"
         },
         "cluster_two": {
+          "mode": "sniff",
           "seeds": [
             "127.0.0.1:9301"
           ],
           "transport.compress": false
+        },
+        "cluster_three": {
+          "mode": "proxy",
+          "proxy_address": "127.0.0.1:9302",
+          "transport.compress": true
         }
       }
     }
@@ -171,7 +215,7 @@ NOTE: When the compression or ping schedule settings change, all the existing
 node connections must close and re-open, which can cause in-flight requests to
 fail.
 
-A remote cluster can be deleted from the cluster settings by setting its seeds and optional settings to `null` :
+A remote cluster can be deleted from the cluster settings by setting its settings to `null` :
 
 [source,console]
 --------------------------------
@@ -181,6 +225,7 @@ PUT _cluster/settings
     "cluster": {
       "remote": {
         "cluster_two": { <1>
+          "mode": null,
           "seeds": null,
           "skip_unavailable": null,
           "transport": {
@@ -201,23 +246,15 @@ PUT _cluster/settings
 [[remote-cluster-settings]]
 === Remote cluster settings
 
-`cluster.remote.connections_per_cluster`::
-
-  The number of gateway nodes to connect to per remote cluster. The default is
-  `3`.
+`cluster.remote.${cluster_alias}.mode`::
+  The mode used for a remote cluster connection. The only supported modes are
+  `sniff` and `proxy`.
 
 `cluster.remote.initial_connect_timeout`::
 
   The time to wait for remote connections to be established when the node
   starts. The default is `30s`.
 
-`cluster.remote.node.attr`::
-
-  A node attribute to filter out nodes that are eligible as a gateway node in
-  the remote cluster. For instance a node can have a node attribute
-  `node.attr.gateway: true` such that only nodes with this attribute will be
-  connected to if `cluster.remote.node.attr` is set to `gateway`.
-
 `cluster.remote.connect`::
 
   By default, any node in the cluster can act as a cross-cluster client and
@@ -251,6 +288,49 @@ PUT _cluster/settings
   Elasticsearch compresses the response. If unset, the global
   `transport.compress` is used as the fallback setting.
 
+[float]
+[[remote-cluster-sniff-settings]]
+=== Remote cluster sniff mode settings
+
+`cluster.remote.${cluster_alias}.seeds`::
+
+  The list of seed nodes used to sniff the remote cluster state.
+
+`cluster.remote.${cluster_alias}.node_connections`::
+
+  The number of gateway nodes to connect to for this remote cluster. The default
+  is `3`.
+
+`cluster.remote.node.attr`::
+
+  A node attribute to filter out nodes that are eligible as a gateway node in
+  the remote cluster. For instance a node can have a node attribute
+  `node.attr.gateway: true` such that only nodes with this attribute will be
+  connected to if `cluster.remote.node.attr` is set to `gateway`.
+
+[float]
+[[remote-cluster-proxy-settings]]
+=== Remote cluster proxy mode settings
+
+`cluster.remote.${cluster_alias}.proxy_address`::
+
+  The address used for all remote connections.
+
+`cluster.remote.${cluster_alias}.proxy_socket_connections`::
+
+  The number of socket connections to open per remote cluster. The default is
+  `18`.
+ifndef::include-xpack[]
+
+`cluster.remote.${cluster_alias}.server_name`::
+
+  An optional hostname string which will be sent in the server_name field of
+  the TLS Server Name Indication extension if
+  <<configuring-tls,{TLS is enabled}>>. The TLS transport will fail to open
+  remote connections if this field is not a valid hostname as defined by the
+  TLS SNI specification.
+endif::[]
+
 [float]
 [[retrieve-remote-clusters-info]]
 === Retrieving remote clusters info