Add support for retries in Reindex API

[DylanGriffith]

When performing a Re-index I have seen several times where the overall reindex fails for transient causes (eg. Node not connected or No search context found for id). When this happens I have the option to try and track down the failed slices and retry those using Manual Slicing. But this is not ideal since it requires manual effort and is error prone. Alternatively if lots failed I may end up preferring to just retry the entire reindex.

It would be good if the Reindex API supported a few more options. For example retries to specify the number of times to retry a failed slice. If it's difficult to tell the difference between transient errors and other errors you could also make it required for the user to specify the retry_errors that should be allowed to retry and anything else should be considered a permanent failure.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Reindex)

[henningandersen]

@DylanGriffith thanks for your interest in Elasticsearch. We have an existing issue to add resiliency to reindex (#42612), which seems to be what you are looking for?

Seeing "Node not connected" would normally signal a network level problem and this may be worth looking into to improve stability until that effort lands in the future.

Also, "No search context found for id" could be an indication of instability (network/node) or that the default scroll timeout is too small for your use case. It might be worth trying with a larger scroll timeout to see if this helps.

Given that we already have this problem registered in #42612, I will go ahead and close this issue.