CAEP: machine deletion phase hooks

Defines a set of annotations that can be applied to a machine which affect the
linear progress of a machine’s lifecycle after a machine has been marked for
deletion.  These annotations are optional and may be applied during machine
creation, sometime after machine creation by a user, or sometime after machine
creation by another controller or application.

Author: michaelgugino

docs/proposals/20200602-machine-deletion-phase-hooks.md

@@ -0,0 +1,297 @@
+---
+title: Machine Deletion Phase Hooks
+authors:
+  - "@michaelgugino"
+reviewers:
+  - "@enxebre"
+  - "@vincepri"
+  - "@detiber"
+  - "@ncdc"
+creation-date: 2020-06-02
+last-updated: 2020-06-02
+status: implementable
+---
+
+# Machine Deletion Phase Hooks
+
+## Table of Contents
+<!-- toc -->
+- [Machine Deletion Phase Hooks](#machine-deletion-phase-hooks)
+  - [Table of Contents](#table-of-contents)
+  - [Glossary](#glossary)
+    - [lifecycle hook](#lifecycle-hook)
+    - [deletion phase](#deletion-phase)
+  - [Summary](#summary)
+  - [Motivation](#motivation)
+    - [Goals](#goals)
+    - [Non-Goals/Future Work](#non-goalsfuture-work)
+  - [Proposal](#proposal)
+    - [User Stories](#user-stories)
+      - [Story 1](#story-1)
+      - [Story 2](#story-2)
+      - [Story 3](#story-3)
+    - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)
+      - [Lifecycle Points](#lifecycle-points)
+        - [pre-drain](#pre-drain)
+        - [pre-term](#pre-term)
+      - [Annotation Form](#annotation-form)
+        - [lifecycle-point](#lifecycle-point)
+        - [hook-name](#hook-name)
+        - [owner (Optional)](#owner-optional)
+        - [Annotation Examples](#annotation-examples)
+      - [Changes to machine-controller](#changes-to-machine-controller)
+        - [Reconciliation](#reconciliation)
+        - [Hook failure](#hook-failure)
+        - [Hook ordering](#hook-ordering)
+      - [Owning Controller Design](#owning-controller-design)
+        - [Owning Controllers must](#owning-controllers-must)
+        - [Owning Controllers may](#owning-controllers-may)
+      - [Determining when to take action](#determining-when-to-take-action)
+        - [Failure Mode](#failure-mode)
+    - [Risks and Mitigations](#risks-and-mitigations)
+  - [Alternatives](#alternatives)
+    - [Custom Machine Controller](#custom-machine-controller)
+    - [Finalizers](#finalizers)
+    - [Status Field](#status-field)
+    - [Spec Field](#spec-field)
+    - [CRDs](#crds)
+  - [Upgrade Strategy](#upgrade-strategy)
+  - [Additional Details](#additional-details)
+<!-- /toc -->
+
+## Glossary
+
+Refer to the [Cluster API Book Glossary](https://cluster-api.sigs.k8s.io/reference/glossary.html).
+
+### lifecycle hook
+A specific point in a machine's reconciliation lifecycle where execution of
+normal machine-controller behavior is paused or modified.
+
+### deletion phase
+Describes when a machine has been marked for deletion but is still present
+in the API.  Various actions happen during this phase, such as draining a node,
+deleting an instance from a cloud provider, and deleting a node object.
+
+## Summary
+
+Defines a set of annotations that can be applied to a machine which affect the
+linear progress of a machine’s lifecycle after a machine has been marked for
+deletion.  These annotations are optional and may be applied during machine
+creation, sometime after machine creation by a user, or sometime after machine
+creation by another controller or application.
+
+## Motivation
+
+Allow custom and 3rd party components to easily interact with a machine or
+related resources while that machine's reconciliation is temporarily paused.
+This pause in reconciliation will allow these custom components to take action
+after a machine has been marked for deletion, but prior to the machine being
+drained and/or associated instance terminated.
+
+### Goals
+
+- Define an initial set of hook points for the deletion phase.
+- Define an initial set and form of related annotations.
+- Define basic expectations for a controller or process that responds to a
+lifecycle hook.
+
+
+### Non-Goals/Future Work
+
+- Create an exhaustive list of hooks; we can add more over time.
+- Create new machine phases
+- Dictate implementation of controllers that respond to the hooks.
+- Implement ordering in the machine-controller.
+- Require anyone use these hooks for normal machine operations, these are
+strictly optional and for custom integrations only.
+
+
+## Proposal
+
+- Utilize annotations to implement lifecycle hooks.
+- Each lifecycle point can have 0 or more hooks.
+- Hooks do not enforce ordering.
+- Hooks found during machine reconciliation effectively pause reconciliation
+until all hooks for that lifecycle point are removed from a machine's annotations.
+
+
+### User Stories
+#### Story 1
+(pre-term) As an operator, I would like to have the ability to perform
+different actions between the time a machine is marked deleted in the api and
+the time the machine is deleted from the cloud.
+
+For example, when replacing a control plane machine, ensure a new control
+plane machine has been successfully created and joined to the cluster before
+removing the instance of the deleted machine. This might be useful in case
+there are disruptions during replacement and we need the disk of the existing
+instance to perform some disaster recovery operation.  This will also prevent
+prolonged periods of having one fewer control plane host in the event the
+replacement instance does not come up in a timely manner.
+
+#### Story 2
+(pre-drain) As an operator, I want the ability to utilize my own draining
+controller instead of the logic built into the machine-controller.  This will
+allow me better flexibility and control over the lifecycle of workloads on each
+node.
+
+#### Story 3
+(pre-drain) As an operator, when I am deleting a control plane machine for
+maintenance reasons, I want to ensure my existing control plane machine is not
+drained until after my replacement comes online.  This will prevent protracted
+periods of a missing control plane member if the replacement machine cannot be
+created in a timely manner.
+
+
+### Implementation Details/Notes/Constraints
+
+For each defined lifecycle point, one or more hooks may be applied as an annotation to the machine object.  These annotations will pause reconciliation of a machine object until all hooks are resolved for that lifecycle point.  The hooks should be managed by an Owning Controller or other external application, or manually created and removed by an administrator.
+
+#### Lifecycle Points
+##### pre-drain
+Hooks defined at this point will prevent the machine-controller from draining a node after the machine-object has been marked for deletion until the hooks are removed.
+##### pre-term
+Hooks defined at this point will prevent the machine-controller from
+removing/terminating the instance in the cloud provider until the hooks are
+removed.
+
+"pre-term" has been chosen over "pre-delete" because "terminate" is more
+easily associated with an instance being removed from the cloud or
+infrastructure, whereas "delete" is ambiguous as to the actual state of the
+machine in its lifecycle.
+
+
+#### Annotation Form
+```
+<lifecycle-point>.hook.machine.cluster-api.x-k8s.io/<hook-name>: <owner/creator>
+```
+
+##### lifecycle-point
+This is the point in the lifecycle of reconciling a machine the annotation will have effect and pause the machine-controller.
+
+##### hook-name
+Each hook should have a unique and descriptive name that describes in 1-3 words what the intent/reason for the hook is.  Each hook name should be unique and managed by a single entity.
+
+##### owner (Optional)
+Some information about who created or is otherwise in charge of managing the annotation.  This might be a controller or a username to indicate an administrator applied the hook directly.
+
+##### Annotation Examples
+pre-drain.hook.machine.cluster-api.x-k8s.io/migrate-important-app: my-app-migration-controller
+
+pre-term.hook.machine.cluster-api.x-k8s.io/preserve-instance: my-instance-replacement-controller
+
+pre-term.hook.machine.cluster-api.x-k8s.io/never-delete: cluster-admin
+
+#### Changes to machine-controller
+The machine-controller should check for the existence of 1 or more hooks at
+specific points (lifecycle-points) during reconciliation.  If a hook matching
+the lifecycle-point is discovered, the machine-controller should stop
+reconciling the machine.
+
+An example of where the pre-drain lifecycle-point might be implemented:
+https://github.com/kubernetes-sigs/cluster-api/blob/30c377c0964efc789ab2f3f7361eb323003a7759/controllers/machine_controller.go#L270
+
+##### Reconciliation
+When an Owning Controller updates the machine, reconciliation will be
+triggered, and the machine will continue reconciling as normal, unless another
+hook is still present; there is no need to 'fail' the reconciliation to
+enforce requeuing.
+
+When all hooks for a given lifecycle-point are removed, reconciliation
+will continue as normal.
+
+##### Hook failure
+The machine-controller should not timeout or otherwise consider the lifecycle
+hook as 'failed.'  Only the Owning Controller may decide to remove a
+particular lifecycle hook to allow the machine-controller to progress past the
+corresponding lifecycle-point.
+
+##### Hook ordering
+The machine-controller will not attempt to enforce any ordering of hooks.
+
+Owning Controllers may devise a dependency-based enforcement mechanism
+via whatever means Owning Controllers determine.  Examples could be
+using CRDs external to the machine-api, gRPC communications, or
+additional annotations on the machine or other objects.
+
+#### Owning Controller Design
+Owning Controller is the component that manages a particular lifecycle hook.
+
+##### Owning Controllers must
+* Watch machine objects and determine when an appropriate action must be taken.
+* After completing the desired hook action, remove the hook annotation.
+##### Owning Controllers may
+* Watch machine objects and add a hook annotation as desired by the operator
+* After completing a hook, set an addition annotation that indicates this
+controller has finished for dependency ordering, or perform some other action
+to signal other Owning Controllers that they may proceed.
+
+#### Determining when to take action
+
+An Owning Controller should watch machines and determine when is the
+best time to take action.
+
+For example, if an Owning Controller manages a lifecycle hook at the
+pre-drain lifecycle-point, then that controller should take action
+immediately after a machine has a DeletionTimestamp or enters the
+"Deleting" phase.
+
+Fine-tuned coordination is not possible at this time; eg, it's not
+possible to execute a pre-term hook only after a node has been drained.
+
+##### Failure Mode
+It is entirely up to the owning controller to determine when it is prudent to
+remove a particular lifecycle hook.  Some controllers may want to 'give up'
+after a certain time period, and others may want to block indefinitely.
+Cluster operators should consider the characteristics of each controller before
+utilizing them in their clusters.
+
+
+### Risks and Mitigations
+
+* Annotation keys must conform to length limits: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set
+* Requires well-behaved controllers and admins to keep things running
+smoothly.  Would be easy to disrupt machines with poor configuration.
+* Troubleshooting problems may increase in complexity, but this is
+mitigated mostly by the fact that these hooks are opt-in.  Operators
+will or should know they are consuming these hooks, but a future proliferation
+of the cluster-api could result in these components being bundled as a
+complete solution that operators just consume.  To this end, we should
+update any troubleshooting guides to check these hook points where possible.
+
+
+## Alternatives
+
+### Custom Machine Controller
+Require advanced users to fork and customize.  This can already be done if someone chooses, so not much of a solution.
+
+### Finalizers
+We define additional finalizers, but this really only implies the deletion lifecycle point.  A misbehaving controller that
+accidentally removes finalizers could have undesireable
+effects.
+
+### Status Field
+Harder for users to modify or set hooks during machine creation.  How would a user remove a hook if a controller that is supposed to remove it is misbehaving?  We’d probably need an annotation like ‘skip-hook-xyz’ or similar and that seems redundant to just using annotations in the first place
+
+### Spec Field
+We probably don’t want other controllers dynamically adding and removing spec fields on an object.  It’s not very declarative to utilize spec fields in that way.
+
+### CRDs
+Seems like we’d need to sync information to and from a CR.  There are different approaches to CRDs (1-to-1 mapping machine to CR, match labels, present/absent vs status fields) that each have their own drawbacks and are more complex to define and configure.
+
+
+## Upgrade Strategy
+
+Nothing defined here should directly impact upgrades other than defining hooks that impact creation/deletion of a machine, generally.
+
+## Additional Details
+
+Fine-tuned timing of hooks is not possible at this time.
+
+In the future, it is possible to implement this timing via additional
+machine phases, or possible "sub-phases" or some other mechanism
+that might be appropriate.  As stated in the non-goals, that is
+not in scope at this time, and could be future work.
+
+<!-- Links -->
+[community meeting]: https://docs.google.com/document/d/1Ys-DOR5UsgbMEeciuG0HOgDQc8kZsaWIWJeKJ1-UfbY