containerd should be using systemd cgroup hierarchy

[randomvariable]

What steps did you take and what happened:
[A clear and concise description on how to REPRODUCE the bug.]

As per https://kubernetes.io/docs/setup/production-environment/container-runtimes/, there should be only be a single cgroup hiearchy running on the OS. We do not configure containerd to run using the systemd hierarchy, such that Kubernetes will have an incorrect view of resource utilisation.

What did you expect to happen:

ContainerD should be configured to use systemd hierarchy

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

ContainerD config.toml should have something along the lines of

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

Environment:

Project (Image Builder for Cluster API, kube-deploy/imagebuilder, konfigadm):

Additional info for Image Builder for Cluster API related issues:

• OS (e.g. from /etc/os-release, or cmd /c ver):
• Packer Version:
• Packer Provider:
• Ansible Version:
• Cluster-api version (if using):
• Kubernetes version: (use kubectl version):

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[neolit123]

just appending:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

at the end of the config should work too.
(but this assumes that there is [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] key already)

[neolit123]

the tricky part is that once this is done the following KubeletConfiguration must be passed to kubeadm:

...
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: "systemd"

@fabriziopandini @vincepri @detiber for comments on CAPI.

[timothysc]

@codenrhoden - We need to fix this.

[codenrhoden]

the tricky part is that once this is done the following KubeletConfiguration must be passed to kubeadm:

So this sounds like we have to coordinate the fix with kubeadm at the same time? Or if we have the Cgroup setup through systemd for future images, and kubeadm does not pass the needed KubeletConfiguration, will everything still work as before?

[neolit123]

So this sounds like we have to coordinate the fix with kubeadm at the same time? Or if we have the Cgroup setup through systemd for future images, and kubeadm does not pass the needed KubeletConfiguration, will everything still work as before?

if we add the systemd on the image side and the kubelet is not configured with systemd it will break. the default value in the kubelet is cgroupfs which is sane as the kubelet may not be run with systemd (as a service manager).

there are a couple of options:

1. default the value to systemd in the KubeletConfiguration the kubeadm creates, which is a breaking change to the wider public.
2. apply the KubletConfigurtion with systemd on a higher layer - e.g. CAPI.

after some though, i'm leaning towards 2 but would appreciate more opinions here.

[randomvariable]

If we configure it with Cluster API:
More likely we need to co-ordinate with Cluster API provider template authors in order to be able to passdown the cgroupDriver setting. And we don't want to default it in Cluster API Bootstrap Provider Kubeadm unless it's OS aware because of Windows?

Is there a last option in that we set the deprecated cgroup driver flag in /etc/sysconfig/kubelet such that CAPI doesn't need to do it ?

[neolit123]

More likely we need to co-ordinate with Cluster API provider template authors in order to be able to passdown the cgroupDriver setting.

its quite unfortunate but all users should do this setting unless kubeadm does it by default and given it runs the kubelet with systemd. which creates this period of breaking users that were using cgroupfs on the CR side.

And we don't want to default it in Cluster API Bootstrap Provider Kubeadm unless it's OS aware because of Windows?

cgroup drivers should be completely ignored on Windows as there is no such thing there, but we found a bug where the kubelet errors on Windows if the driver reported by docker info differs:
kubernetes/kubernetes#97764

Is there a last option in that we set the deprecated cgroup driver flag in /etc/sysconfig/kubelet such that CAPI doesn't need to do it ?

the kubelet's --cgroup-driver flag is being removed soon, so this is not going to work well long term. the config file doesn't support similar overrides.

[neolit123]

opening the wider discussion:
kubernetes/kubeadm#2376

[oldthreefeng]

@neolit123 for containerd

$ containerd --version
containerd github.com/containerd/containerd v1.3.9 ea765aba0d05254012b0b9e595e995c09186427f

I found default systemd_cgroup section is not [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]

$ containerd config  default  
...
  [plugins."io.containerd.grpc.v1.cri"]
    systemd_cgroup = false
    disable_cgroup = false
...

[randomvariable]

the docs may be out of date.

In any case, after some discussion, we've settled on the following:

• Let's make the containerd change
• Set the cgroup driver flag for kubelet in /etc/sysconfig/kubelet. Put a TODO that we need to not set that flag in whatever version Kubernetes removes that flag.
• It is up to CAPI to sort out setting the kubelet component config for future Kubernetes versions.

[neolit123]

The field in the current docs works for me. Without it the node failed. Is this a recent containerd change?

[oldthreefeng]

@neolit123 Im sorry to confused you.
You are definitely right. I am confused by the section of .

  [plugins."io.containerd.grpc.v1.cri"]
    systemd_cgroup = false

forget it

[fabriziopandini]

I'm trying to better understand what is the expected UX in CAPI after this change lands in the image builder

AFAIK currently it is not possible to change the KubeletConfiguration for the users (see kubernetes-sigs/cluster-api#1584), so I assume we should still rely on ExtraArgs, which is not ideal.

Few options:

• The user should set the --cgroup-driver extra arg before adding a new machine or before triggering rollouts
• CAPBK should set the --cgroup-driver extra arg if missing

Also, after the change is implemented in the image builder is there any expectation on how the change would impact the images published by the providers e.g.

• It will impact only new images (how do we define new images)
• All the existing images will be regnerated
• ???

[neolit123]

IMO, the images should be versioned with k8s.