[Test] Additional logging for RemoteClusterClientTests

[cbuescher]

Use RemoteClusterConnectionTests#startTransport instead of
MockTransportService.createNewService to start the local node transport service.
This implicitely calls newService.start() and newService.acceptIncomingRequests()
but potentially doesn't run into the same kind of deadlock situation that we occasionally saw
on recent test runs.

Closes #41745

[elasticmachine]

Pinging @elastic/es-search

[cbuescher]

@original-brownbear would you mind taking a look at this since we already talked about the problem on the issue? If not let me know if I should ask somebody else to review. I'm not sure this fixed things but it looks like it should do the same as before, maybe the order of execution is somewhat safer than in the existing test.

[original-brownbear]

@cbuescher hmm, but the stacktrace we have suggests that the latch was never counted down?
Looking at the code here:

                service.start();
                service.acceptIncomingRequests();
                RemoteClusterService remoteClusterService = service.getRemoteClusterService();
                assertTrue(remoteClusterService.isRemoteNodeConnected("test", remoteNode));

this currently fails on the last line (the assertTrue). So acceptIncomingRequests was called for sure?
=> I'm not sure how this solves issue. Maybe instead add some info level logging before and after the explicit service.acceptIncomingRequests(); in the existing code so we can see the timing of when that is called relative to the failure stack traces? (maybe we're hanging before that, that seems like a likely explanation?)

[original-brownbear]

I'm also wondering if this might be explained by some port collision, now resolved in #43983 (I can't immediately see how, but I think if we added the suggested info level logging and waited for the next failure then we can rule out it was a port collision ... because the next failure shouldn't come)

[cbuescher]

So acceptIncomingRequests was called for sure?

My guess was that this might happen after the test timeout though. But I will revert and add the logging you suggested to get more information on this question. If this doesn't fail again soon we can also maybe close and blame it on #43983? But I'll add the logging instead of the changes I made earlier.

[original-brownbear]

@cbuescher

If this doesn't fail again soon we can also maybe close and blame it on #43983? But I'll add the logging instead of the changes I made earlier.

Thanks, I like that plan :)

[original-brownbear]

LGTM

[cbuescher]

@elasticmachine run elasticsearch-ci/packaging-sample

[cbuescher]

@elasticmachine run elasticsearch-ci/bwc

[cbuescher]

@elasticmachine update branch