Drain nodes when they terminate

[grosser]

When the ASG scales down the node should be drained and not just terminated (killing pods / not respecting poddisruptionbudget)

Either via "Amazon EC2 Auto Scaling Lifecycle Hooks" (up to 60 min) or with a termination script https://github.com/miglen/aws/blob/master/ec2/run-script-on-ec2-instance-termination.md (2 min max, so will be tight)

Flow could be Scale-Down -> SQS -> Drainer or Scale-Down -> SQS -> Node status -> https://github.com/planetlabs/draino

We might be able to contribute this, but need some "yes that's a good idea" / "yes we want this" first :)

[pracucci]

I would personally suggest with the easiest approach, which looks the termination script and eventually iterate on it over the time.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[grosser]

/remove-lifecycle stale

…

On Thu, Sep 5, 2019 at 10:26 AM fejta-bot ***@***.***> wrote: Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Send feedback to sig-testing, kubernetes/test-infra and/or fejta <https://github.com/fejta>. /lifecycle stale — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7119?email_source=notifications&email_token=AAACYZ7OEMEBFG2QE6QQ2NDQIE6MJA5CNFSM4HVYMZNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6ABAWQ#issuecomment-528486490>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAACYZ5WORU3IMVVNVPFKT3QIE6MJANCNFSM4HVYMZNA> .

[so0k]

I did a detailed write up on setting up LCH / SQS / node-drainer here

https://mofoseng.github.io/posts/eks-migration/#node-maintenance-autoscaling-and-lifecyclehooks

I compared kube-aws drainer (which has a single replica deployment updating a configmap and ds on each node grepping the configmap... ) to a pure AWS lambda based approach and this SQS approach seemed the most robust

it's not kops specific but it may help adoption (I moved company and now I'm back on kops after migrating out of it at my last company)

[grosser]

FYI We use a similar workflow but instead of a configmap set a label on the node to mark it as draining ... Also looking at moving to a single deployment to have less pods flying around.

…

On Tue, Sep 10, 2019, 8:29 PM so0k ***@***.***> wrote: I did a detailed write up on setting up LCH / SQS / node-drainer here https://mofoseng.github.io/posts/eks-migration/#node-maintenance-autoscaling-and-lifecyclehooks I compared kube-aws drainer (which has a single replica deployment updating a configmap and ds on each node grepping the configmap... ) to a pure AWS lambda based approach and this SQS approach seemed the most robust it's not kops specific but it may help adoption (I moved company and now I'm back on kops after migrating out of it at my last company) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7119?email_source=notifications&email_token=AAACYZ3LZ4NMWBVEGCUL2BDQJBQYJA5CNFSM4HVYMZNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6NEM2Y#issuecomment-530204267>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAACYZ62FXSQSK2VO333W7TQJBQYJANCNFSM4HVYMZNA> .

[so0k]

@grosser - how did you manage LCH creation for kops currently? do you generate TF and write config around that? Did you fork kops to add that functionality?

[grosser]

we are not doing that for kops yet, just our old clusters that we create with cloudformation, still in the progress of figuring that out :(

…

On Thu, Sep 12, 2019 at 8:01 PM so0k ***@***.***> wrote: @grosser <https://github.com/grosser> - how did you manage LCH creation for kops currently? do you generate TF and write config around that? Did you fork kops to add that functionality? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7119?email_source=notifications&email_token=AAACYZ3L2FVN4SSHFTDBNR3QJL7A3A5CNFSM4HVYMZNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6T2LUQ#issuecomment-531080658>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAACYZ2PPCAIFGCTUDWQNULQJL7A3ANCNFSM4HVYMZNA> .

[so0k]

sorry for off-topic

I'm also trying to taint spot instances in a mixedInstancePolicy - for this I see 2 approaches (without modifying kops):

1. A Kubelet systemd service unit drop-in as a hook or an asset which adds --register-with-taints / --node-labels to the $DAEMON_ARGS based on aws ec2 describe-instances --instance-ids ${iid} --query 'Reservations[0].Instances[0].InstanceLifecycle' output
2. A DS similar iameli/kube-node-labeller to set taints and labels

The difference is that option 1 would ensure taints are set before node registration, option 2 would take effect at some point in time later (maybe after some workloads which should not tolerate the taint have already started running on the tainted node...)

with EKS bootstrap it is quite simple to ensure nodes are labeled / tainted properly before they register with the API (and kops only supports labels/taints at the iG level, not at the per node level)

[so0k]

For now we implemented option 2: https://github.com/compareasiagroup/kube-node-lifecycle-labeller

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[grosser]

/remove-lifecycle stale