Put nodes into maintenance mode in preStop hook (#378)

* Put nodes into maintenance mode in preStop hook

This commit causes nodes to go into maintenance mode before the
controller runtime terminates the rabbitmq-server process on a pod stop.
This allows for a more controlled shutdown of pods in a cluster during
rolling restart, or upgrade.

Using github.com/Vanlightly/RabbitTestTool, I was able to show that with
these changes, no data loss, or serious periods of cluster
unavailability, were caused during a rolling restart. However, this is
the case without these changes; I attempted to find a demonstrable case
where this change leads to an improvement in rolling restart
reliability, however was not able to improve this beyond its current
reliability.

I still put this PR forward for review, as the core team recommend that
nodes are put into maintenance mode before being stopped, to allow more
predictable transfer of responsibility to other nodes. The
`rabbitmq-upgrade drain` command is also able to be expanded in the
future if there are further improvements that can be made in the future.

This closes #320.

* Update minimum RMQ version to 3.8.8

Maintenance mode is not available in previous versions of RabbitMQ

Author: coro

README.md

@@ -39,7 +39,7 @@ under `site/kubernetes`.
 
 ## Supported Versions
 
-The operator deploys RabbitMQ `3.8.8` by default, and supports versions from `3.8.4` upwards. The operator requires Kubernetes `1.16` or newer.
+The operator deploys RabbitMQ `3.8.8` by default, and supports versions from `3.8.8` upwards. The operator requires Kubernetes `1.16` or newer.
 
 ## Versioning
 

internal/resource/statefulset.go

@@ -620,7 +620,8 @@ func (builder *StatefulSetBuilder) podTemplateSpec(annotations, labels map[strin
 								Command: []string{"/bin/bash", "-c",
 									fmt.Sprintf("if [ ! -z \"$(cat /etc/pod-info/%s)\" ]; then exit 0; fi;", DeletionMarker) +
 										fmt.Sprintf(" rabbitmq-upgrade await_online_quorum_plus_one -t %d;"+
-											" rabbitmq-upgrade await_online_synchronized_mirror -t %d", defaultGracePeriodTimeoutSeconds, defaultGracePeriodTimeoutSeconds),
+											" rabbitmq-upgrade await_online_synchronized_mirror -t %d;"+
+											" rabbitmq-upgrade drain -t %d", defaultGracePeriodTimeoutSeconds, defaultGracePeriodTimeoutSeconds, defaultGracePeriodTimeoutSeconds),
 								},
 							},
 						},

internal/resource/statefulset_test.go

@@ -1006,7 +1006,7 @@ var _ = Describe("StatefulSet", func() {
 			stsBuilder := builder.StatefulSet()
 			Expect(stsBuilder.Update(statefulSet)).To(Succeed())
 
-			expectedPreStopCommand := []string{"/bin/bash", "-c", "if [ ! -z \"$(cat /etc/pod-info/skipPreStopChecks)\" ]; then exit 0; fi; rabbitmq-upgrade await_online_quorum_plus_one -t 604800; rabbitmq-upgrade await_online_synchronized_mirror -t 604800"}
+			expectedPreStopCommand := []string{"/bin/bash", "-c", "if [ ! -z \"$(cat /etc/pod-info/skipPreStopChecks)\" ]; then exit 0; fi; rabbitmq-upgrade await_online_quorum_plus_one -t 604800; rabbitmq-upgrade await_online_synchronized_mirror -t 604800; rabbitmq-upgrade drain -t 604800"}
 
 			Expect(statefulSet.Spec.Template.Spec.Containers[0].Lifecycle.PreStop.Exec.Command).To(Equal(expectedPreStopCommand))
 		})