controllers-and-reconciliation.md

Controllers and Reconciliation

Right now, you can create objects with your API types, but those objects don't make any impact on your mailgun infrastructure. Let's fix that by implementing controllers and reconciliation for your API objects.

From the kubebuilder book:

Controllers are the core of Kubernetes, and of any operator.

It’s a controller’s job to ensure that, for any given object, the actual state of the world (both the cluster state, and potentially external state like running containers for Kubelet or loadbalancers for a cloud provider) matches the desired state in the object. Each controller focuses on one root Kind, but may interact with other Kinds.

We call this process reconciling.

Also in this case, controllers and reconcilers generated by Kubebuilder are just a shell. It is up to you to fill it with the actual implementation.

Let's see the Code

Kubebuilder has created our first controller in controllers/mailguncluster_controller.go. Let's take a look at what got generated:

// MailgunClusterReconciler reconciles a MailgunCluster object
type MailgunClusterReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=infrastructure.cluster.x-k8s.io,resources=mailgunclusters,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=infrastructure.cluster.x-k8s.io,resources=mailgunclusters/status,verbs=get;update;patch

func (r *MailgunClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = logf.FromContext(ctx)

	// TODO(user): your logic here

	return ctrl.Result{}, nil
}

RBAC Roles

Before looking at (add) your logic here, lets focus for a moment on the markers before the Reconcile func.

The // +kubebuilder... lines tell kubebuilder to generate RBAC roles so the manager we're writing can access its own managed resources. These should already exist in controllers/mailguncluster_controller.go:

// +kubebuilder:rbac:groups=infrastructure.cluster.x-k8s.io,resources=mailgunclusters,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=infrastructure.cluster.x-k8s.io,resources=mailgunclusters/status,verbs=get;update;patch

We also need to add rules that will let it retrieve (but not modify) Cluster objects. So we'll add another annotation for that, right below the other lines:

// +kubebuilder:rbac:groups=cluster.x-k8s.io,resources=clusters;clusters/status,verbs=get;list;watch

Make sure to add this annotation to MailgunClusterReconciler.

Also, for our MailgunMachineReconciler, access to Cluster API Machine object is needed, so you must add this annotation in controllers/mailgunmachine_controller.go:

// +kubebuilder:rbac:groups=cluster.x-k8s.io,resources=machines;machines/status,verbs=get;list;watch

Regenerate the RBAC roles after you are done:

make manifests

Reconciliation

Let's focus on the MailgunClusterReconciler struct first.

First, a word of warning: no guarantees are made about parallel access, both on one machine or multiple machines. That means you should not store any important state in memory: if you need it, write it into a Kubernetes object and store it.

We're going to be sending mail, so let's add a few extra fields:

// MailgunClusterReconciler reconciles a MailgunCluster object
type MailgunClusterReconciler struct {
	client.Client
	Scheme *runtime.Scheme
	Mailgun   mailgun.Mailgun
	Recipient string
}

Now it's time for our Reconcile function. Reconcile is only passed a name, not an object, so let's retrieve ours.

Here's a naive example:

func (r *MailgunClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	ctx := context.Background()
	_ = ctrl.LoggerFrom(ctx)

	var cluster infrav1.MailgunCluster
	if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
		return ctrl.Result{}, err
	}

	return ctrl.Result{}, nil
}

By returning an error, you request that our controller will get Reconcile() called again. That may not always be what you want - what if the object's been deleted? So let's check that:

    var mailgunCluster infrav1.MailgunCluster
    if err := r.Get(ctx, req.NamespacedName, &mailgunCluster); err != nil {
        // 	import apierrors "k8s.io/apimachinery/pkg/api/errors"
        if apierrors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }

Now, if this were any old kubebuilder project you'd be done, but in our case you have one more object to retrieve. While we defined our own cluster object (MailGunCluster) that represents all the infrastructure provider specific details for our cluster, we also need to retrieve the upstream Cluster object that is defined by Cluster API itself. Luckily, cluster API provides a helper for us.

First, you'll need to import the cluster-api package into our project if you haven't done so yet:

# In your Mailgun repository's root directory
go get sigs.k8s.io/cluster-api
go mod tidy

Now we can add in a call to the GetOwnerCluster function to retrieve the cluster object:

    // import sigs.k8s.io/cluster-api/util
    cluster, err := util.GetOwnerCluster(ctx, r.Client, mailgunCluster.ObjectMeta)
    if err != nil {
        return ctrl.Result{}, err
    }

If our cluster was just created, the Cluster API controller may not have set the ownership reference on our object yet, so we'll have to return here and wait to do more with our cluster object until then. We can leave a log message noting that we're waiting for the main Cluster API controller to set the ownership reference. Here's what our Reconcile() function looks like now:

func (r *MailgunClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // You'll eventually get rid of this and use a context passed in from your main.go
	ctx := context.Background() 

    // We change the _ to `log` since we're going to log something now
	log = ctrl.LoggerFrom(ctx)

    var mailgunCluster infrav1.MailgunCluster
    if err := r.Get(ctx, req.NamespacedName, &mailgunCluster); err != nil {
        // import apierrors "k8s.io/apimachinery/pkg/api/errors"
        if apierrors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }

    // import sigs.k8s.io/cluster-api/util
    cluster, err := util.GetOwnerCluster(ctx, r.Client, mailgunCluster.ObjectMeta)
    if err != nil {
        return ctrl.Result{}, err
    }

	if cluster == nil {
		log.Info("Waiting for Cluster Controller to set OwnerRef on MailGunCluster")
		return ctrl.Result{}, nil
	}

The fun part

More Documentation: The Kubebuilder Book has some excellent documentation on many things, including how to write good controllers!

Now that you have all the objects you care about, it's time to do something with them! This is where your provider really comes into its own. In our case, let's try sending some mail:

subject := fmt.Sprintf("[%s] New Cluster %s requested", mailgunCluster.Spec.Priority, cluster.Name)
body := fmt.Sprintf("Hello! One cluster please.\n\n%s\n", mailgunCluster.Spec.Request)

msg := r.mailgun.NewMessage(mailgunCluster.Spec.Requester, subject, body, r.Recipient)
_, _, err = r.Mailgun.Send(msg)
if err != nil {
    return ctrl.Result{}, err
}

Idempotency

But wait, this isn't quite right. Reconcile() gets called periodically for updates, and any time any updates are made. That would mean we're potentially sending an email every few minutes! This is an important thing about controllers: they need to be idempotent. This means a controller must be able to repeat actions on the same inputs without changing the effect of those actions.

So in our case, we'll store the result of sending a message, and then check to see if we've sent one before.

    if mailgunCluster.Status.MessageID != nil {
        // We already sent a message, so skip reconciliation
        return ctrl.Result{}, nil
    }
    
    subject := fmt.Sprintf("[%s] New Cluster %s requested", mailgunCluster.Spec.Priority, cluster.Name)
    body := fmt.Sprintf("Hello! One cluster please.\n\n%s\n", mailgunCluster.Spec.Request)
    
    msg := r.Mailgun.NewMessage(mailgunCluster.Spec.Requester, subject, body, r.Recipient)
    _, msgID, err := r.Mailgun.Send(msg)
    if err != nil {
        return ctrl.Result{}, err
    }
    
    // patch from sigs.k8s.io/cluster-api/util/patch
    helper, err := patch.NewHelper(&mailgunCluster, r.Client)
    if err != nil {
        return ctrl.Result{}, err
    }
    mailgunCluster.Status.MessageID = &msgID
    if err := helper.Patch(ctx, &mailgunCluster); err != nil {
        return ctrl.Result{}, errors.Wrapf(err, "couldn't patch cluster %q", mailgunCluster.Name)
    }
    
    return ctrl.Result{}, nil

A note about the status

Usually, the Status field should only be values that can be computed from existing state. Things like whether a machine is running can be retrieved from an API, and cluster status can be queried by a healthcheck. The message ID is ephemeral, so it should properly go in the Spec part of the object. Anything that can't be recreated, either with some sort of deterministic generation method or by querying/observing actual state, needs to be in Spec. This is to support proper disaster recovery of resources. If you have a backup of your cluster and you want to restore it, Kubernetes doesn't let you restore both spec & status together.

We use the MessageID as a Status here to illustrate how one might issue status updates in a real application.

Update main.go

Since you added fields to the MailgunClusterReconciler, it is now required to update main.go to set those fields when our reconciler is initialized.

Right now, it probably looks like this:

    if err = (&controllers.MailgunClusterReconciler{
        Client: mgr.GetClient(),
        Scheme: mgr.GetScheme(),
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "Unable to create controller", "controller", "MailgunCluster")
        os.Exit(1)
    }

Let's add our configuration. We're going to use environment variables for this:

    domain := os.Getenv("MAILGUN_DOMAIN")
    if domain == "" {
        setupLog.Info("missing required env MAILGUN_DOMAIN")
        os.Exit(1)
    }
    
    apiKey := os.Getenv("MAILGUN_API_KEY")
    if apiKey == "" {
        setupLog.Info("missing required env MAILGUN_API_KEY")
        os.Exit(1)
    }
    
    recipient := os.Getenv("MAIL_RECIPIENT")
    if recipient == "" {
        setupLog.Info("missing required env MAIL_RECIPIENT")
        os.Exit(1)
    }
    
    mg := mailgun.NewMailgun(domain, apiKey)
    
    if err = (&controllers.MailgunClusterReconciler{
        Client:    mgr.GetClient(),
        Scheme: mgr.GetScheme(),
        Mailgun:   mg,
        Recipient: recipient,
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "Unable to create controller", "controller", "MailgunCluster")
        os.Exit(1)
    }

If you have some other state, you'll want to initialize it here!