More Efficient Ordering of Shard Upload Execution (#42791)

[original-brownbear]

Inspired by #39657 and resolving the issue raised in that PR:

The problem currently is that we are uploading all segments in a shard sequentially per shard and only parallelize snapshotting across shards but not within shards.
This PR adjusts the logic towards parallel segment uploads:

• Instead of starting all shard uploads in parallel and without any order, run segment uploads shard by shard.
• Only acquire index commit when it's actually required (after determining the current state of a shard in the repository, not before that).
  • @DaveCTurner pointed out that going sequentially in determining the segments to upload shard by shard increases the timespan between index commits. This is true for the case pf having more snapshot threads than shards to be snapshotted. On the other hand if there's more shards to be snapshotted on a node than snapshot threads it's the other way around. Currently, in this situation once the snapshot thread pool is fully occupied a shard will have to wait for another shard to completely finish its snapshot before taking the index commit (which could be many minutes!) while now the time span between the index commits is likely never more than a few seconds apart (biggest difference between two index commit times is basically equal to the time it takes all the shard folder listing operations to finish)
• Release index commit asap (with this change we generally way reduce the amount of time we hold on to an index commit in cases of uploading multiple shards with multiple segments)

Backport of #42791 and #46208

[elasticmachine]

Pinging @elastic/es-distributed