Prefer nodes that previously held primary shard for primary shard allocation

[ywelsch]

Allocation ids are currently used during primary shard allocation to decide which nodes have non-stale copies of the shard data. Instead of just having a set of allocation ids representing the shard copies that have been last active, we can order this set by putting allocation ids of active primaries first. This allows the primary shard allocator to prefer allocation on nodes that previously held the active primary shard.

Relates to #14739

[bleskes]

Thanks @ywelsch . I was originally thinking of a simpler approach (at least, I think so) - extend NodeGatewayStartedShards to include the primary flag, which we store and use that. This means the change is limited to PrimaryShardAllocator. I like the simplicity of having a Set of active allocation ID, no preference, no ordering.

[ywelsch]

My concern for relying on ShardStateMetaData is that it can be out-of-sync with cluster state. I agree though that for this specific use case (which is just a soft preference) perfect synchrony does not matter and that your approach better scopes the change.

[ywelsch]

@bleskes I updated the PR. can you have a look?

[bleskes]

LGTM. Love the stats +81 -47 (most of it is tests).