Set topologySpreadConstraints by default

[mkuratczyk]

topologySpreadConstraints can be configured in such a way that the StatefulSet will be spread across multiple AZs if they are available but will also deploy to a single-AZ cluster such as kind. Therefore, we can just set the constraints by default as a sane default for most deployments. Ultimately we should achieve something like this:

topologySpreadConstraints:
- maxSkew: 1
   topologyKey: zone
   whenUnsatisfiable: ScheduleAnyway
   labelSelector:
     matchLabels:
       app: rabbitmq

[ChunyiLyu]

topologySpreadConstraints was alpha in both k8s 1.17 and 1.16 and should be enabled by feature gate. It went beta in 1.18.

In my quick experiment, I was able to use topologySpreadConstraints in and 1.17 k8s cluster without enabling the feature gate 🤔 However, the puzzling part is that the podTemplate did not show the topologySpreadConstraints that I have provided. I will look into this more.

[ChunyiLyu]

Found a bug while testing it, and reported in kubernetes/kubernetes

Version compatibility
topologySpreadConstraints is alpha in 1.17, and can only be enabled by feature gate. Providing topologySpreadConstraints in podSpec without enabling the feature gate means that the api server will strip the definition from spec. It won't take effect and will still deploy. It's beta in 1.18 and GA in 1.19, and can be used without enabling the feature gate.

Label
It relies on the topologyKey to be defined for it to work. We can use one of the popular label topology.kubernetes.io/zone doc. This zone label is automatically set by kubelet if the cloud provider provides the zone information, which the vSphere provide does

Caveat

The behavior of whenUnsatisfiable: ScheduleAnyways is different from k8s 1.18 and 1.19 . Details here

There is a bug with whenUnsatisfiable: DoNotSchedule, where even if the topologyKey does not match any in any nodes, pods get scheduled anyways. Bug

[Huang-Wei]

In my quick experiment, I was able to use topologySpreadConstraints in and 1.17 k8s cluster without enabling the feature gate 🤔 However, the puzzling part is that the podTemplate did not show the topologySpreadConstraints that I have provided. I will look into this more.

That's as expected. The featuregate is enforced in both apiserver and scheduler side. If it's not enabled in apiserver side, apiserver will drop the topologySpreadConstraints field.

[Huang-Wei]

Therefore, we can just set the constraints by default as a sane default for most deployments.

Correc. And there is an upcoming beta feature called DefaultTopologySpread in 1.20, which is aimed to help cluster admin to apply a default global behavior. Ref: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/1258-default-pod-topology-spread/README.md