Skip to content

Add Pod Anti Affinity #489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Feb 21, 2019
Merged

Add Pod Anti Affinity #489

merged 8 commits into from
Feb 21, 2019

Conversation

teutonet
Copy link
Contributor

Add pod anti affinity to enforce that postgres pods does not spawn on the same k8s node.

@coveralls
Copy link

coveralls commented Feb 15, 2019

Coverage Status

Coverage remained the same at 23.705% when pulling 986d3ab on teutostack:pr_pod_affinity into 2e9b653 on zalando-incubator:master.

@redbaron
Copy link
Contributor

related #479

@redbaron
Copy link
Contributor

In some cases it is OK to have 2 PG pods on the same node, for instance when testing switchover/failover

@teutonet
Copy link
Contributor Author

@redbaron You're right with #479, but it is a big change and it will take some time to get it done. Are there any timelines regarding this issue?

For production DBs it would be a worst case scenario to have 2 PG pods on the same node, once the node dies.

@Jan-M
Copy link
Member

Jan-M commented Feb 15, 2019

This def. needs a toggle, maybe incl. a toggle per cluster.

While initially this may sound bad for availability it all depends.

In our K8S infrastructure on AWS this is also largely not happening for a single Postgres cluster as pods are bound to different AZs via the different EBS volumes.

But if your storage is EBS it may sometimes in a single AZ not matter so much if pods are on the same host.

Regarding #479 I am looking forward to ideas. In our initial impl. in Python this was somehow neater having the statefulset spec just injected as a String template.

@teutonet
Copy link
Contributor Author

@Jan-M I think it's a good idea to have a switch in the cluster config to enable pod anti affinity and a addtional property to set the pod affinity TopologyKey, default could be kubernetes.io/hostname.

 * use `cluster_labels` for pod anti affinity
 * use `node_readiness_label` when defined
 * added property `enable_pod_antiaffinity` to operator config
 * added property `pod_antiaffinity_topology_key`
@teutonet
Copy link
Contributor Author

@Jan-M Switch for pod anti affinity is implemented as well as the TopologyKey.

Copy link
Member

@sdudoladov sdudoladov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Could you also update the admin docs and the operator params' reference to mention the new option ?

  2. pls also propagate the option to/from the CRD-based operator configuration

 * use `cluster_labels` for pod anti affinity
 * use `node_readiness_label` when defined
 * added property `enable_pod_antiaffinity` to operator config
 * added property `pod_antiaffinity_topology_key`
@sdudoladov sdudoladov merged commit 26a7fdf into zalando:master Feb 21, 2019
@sdudoladov
Copy link
Member

lgtm. thank you, @ocaner-biz

@FxKu FxKu added this to the v1.2 milestone May 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants