no_such_remote_cluster_exception on freshly created remote cluster.

[dashcheulov]

Elasticsearch version: Version: 7.1.1, Build: default/deb/7a013de/2019-05-23T14:04:00.380842Z, JVM: 12.0.1

Plugins installed: [x-pack]

JVM version: 12.0.1

OS version: Linux 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I added remote cluster and expect to create follower index. Instead, I get no_such_remote_cluster_exception. There are no errors in logs.

Steps to reproduce:

$ curl -u "elastic:$(cat pass)" --cacert /usr/share/elasticsearch/ca.crt https://127.0.0.1:9200/_remote/info?pretty
{
  "fsn" : {
    "seeds" : [
      "elasticsearch-prod-fsn-01.onde.in:9301"
    ],
    "connected" : true,
    "num_nodes_connected" : 3,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false
  }
}

$ curl -u "elastic:$(cat pass)" --cacert /usr/share/elasticsearch/ca.crt -XPUT -H "Content-Type: application/json" 'https://127.0.0.1:9200/onde.pay-gate-2021.09.13/_ccr/follow?wait_for_active_shards=1&pretty' -d '{"remote_cluster": "fsn", "leader_index" : "onde.pay-gate-2021.09.13"}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "no_such_remote_cluster_exception",
        "reason" : "no such remote cluster: [fsn]"
      }
    ],
    "type" : "no_such_remote_cluster_exception",
    "reason" : "no such remote cluster: [fsn]"
  },
  "status" : 404
}

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

This bug report is for 7.1.1 which is almost a year past EOL. Does this also happen on any supported version @dashcheulov?

Would you also share the output of GET _nodes from your local cluster?

Would you add the ?error_trace query parameter to the PUT /_ccr/follow command and try again? That will give us some more information about what component is failing to find the remote cluster.

Finally, please would you share the server logs from the local cluster?

[dashcheulov]

Hi Dave, thanks for reply. I'll try on another version later. For now I have

elasticsearch-prod-hel-01:~# curl -u "elastic:$(cat pass)" --cacert /usr/share/elasticsearch/ca.crt -XPUT -H "Content-Type: application/json" 'https://127.0.0.1:9200/onde.pay-gate-2021.09.22/_ccr/follow?wait_for_active_shards=1&pretty&error_trace' -d '{"remote_cluster": "fsn", "leader_index" : "onde.pay-gate-2021.09.22"}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "no_such_remote_cluster_exception",
        "reason" : "no such remote cluster: [fsn]",
        "stack_trace" : "NoSuchRemoteClusterException[no such remote cluster: [fsn]]\n\tat org.elasticsearch.transport.RemoteClusterService.getRemoteClusterClient(RemoteClusterService.java:524)\n\tat org.elasticsearch.client.node.NodeClient.getRemoteClusterClient(NodeClient.java:123)\n\tat org.elasticsearch.xpack.ccr.action.TransportPutFollowAction.masterOperation(TransportPutFollowAction.java:107)\n\tat org.elasticsearch.xpack.ccr.action.TransportPutFollowAction.masterOperation(TransportPutFollowAction.java:47)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:127)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:205)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:161)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:138)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:341)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)\n\tat org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)\n\tat org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:342)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:319)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:260)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:579)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:489)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:281)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:550)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:248)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:312)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:244)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:247)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:273)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:270)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:258)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:224)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:190)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:229)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:192)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:172)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:158)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:107)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121)\n\tat org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:79)\n\tat org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:74)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:300)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:341)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)\n\tat org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)\n\tat org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:342)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:319)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:260)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:579)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:489)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:281)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:550)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:248)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:312)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:244)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:247)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:273)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:270)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:258)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:224)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:190)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:229)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:192)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1036)\n\tat org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:922)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)\n\tat java.lang.Thread.run(Thread.java:835)\n"
      }
    ],
    "type" : "no_such_remote_cluster_exception",
    "reason" : "no such remote cluster: [fsn]",
    "stack_trace" : "NoSuchRemoteClusterException[no such remote cluster: [fsn]]\n\tat org.elasticsearch.transport.RemoteClusterService.getRemoteClusterClient(RemoteClusterService.java:524)\n\tat org.elasticsearch.client.node.NodeClient.getRemoteClusterClient(NodeClient.java:123)\n\tat org.elasticsearch.xpack.ccr.action.TransportPutFollowAction.masterOperation(TransportPutFollowAction.java:107)\n\tat org.elasticsearch.xpack.ccr.action.TransportPutFollowAction.masterOperation(TransportPutFollowAction.java:47)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:127)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:205)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:161)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:138)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:341)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)\n\tat org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)\n\tat org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:342)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:319)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:260)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:579)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:489)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:281)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:550)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:248)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:312)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:244)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:247)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:273)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:270)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:258)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:224)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:190)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:229)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:192)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:172)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:158)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:107)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121)\n\tat org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:79)\n\tat org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:74)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:300)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:341)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)\n\tat org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)\n\tat org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:336)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:84)\n\tat java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:84)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:117)\n\tat org.elasticsearch.action.StepListener.onResponse(StepListener.java:62)\n\tat org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:342)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:319)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:260)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:604)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:579)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:489)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:281)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:550)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:248)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:641)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:312)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:244)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:247)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:639)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:273)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:297)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:270)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:258)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:224)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:190)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:229)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:192)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1036)\n\tat org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:922)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)\n\tat java.lang.Thread.run(Thread.java:835)\n"
  },
  "status" : 404
}

es-logs-hel.tar.gz

[DaveCTurner]

No news about whether it reproduces on a recent version so I am closing this.