Shutting down a node stops the transport layer despite of ongoing indexing operations

[bleskes]

The current shutdown order is as follows:

        injector.getInstance(RestController.class).stop();
        injector.getInstance(TransportService.class).stop();

        for (Class<? extends LifecycleComponent> plugin : pluginsService.services()) {
            injector.getInstance(plugin).stop();
        }
        // we should stop this last since it waits for resources to get released
        // if we had scroll searchers etc or recovery going on we wait for to finish.
        injector.getInstance(IndicesService.class).stop();
        logger.info("stopped");

The means that the transport service is stopped before indices services, meaning that any ongoing write operation in IndicesService are blocked from the network and can't send requests. This can trigger #7572 .

IMHO we should do a best effort in flushing on going operations in IndicesService, block new one from starting and then and only then close the transport service. These days we have mechanics to do so. This does require careful thought.

[joune]

For reference and test case, please see http://stackoverflow.com/questions/31294500/elasticsearch-1-6-seems-to-lose-documents-during-high-availability-test

[bleskes]

I haven't seen a lot of issues of this in recent history. It's also a complex thing to do. Closing for now. If it bubbles up again we can reopen.