-
Notifications
You must be signed in to change notification settings - Fork 442
Add E2E test for cluster class #2235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
598 changes: 598 additions & 0 deletions
598
templates/test/ci/cluster-template-prow-clusterclass-ci-default.yaml
Large diffs are not rendered by default.
Oops, something went wrong.
4,762 changes: 4,762 additions & 0 deletions
4,762
templates/test/ci/cluster-template-prow-topology.yaml
Large diffs are not rendered by default.
Oops, something went wrong.
264 changes: 264 additions & 0 deletions
264
templates/test/ci/prow-clusterclass-ci-default/base.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,264 @@ | ||
| apiVersion: cluster.x-k8s.io/v1beta1 | ||
| kind: ClusterClass | ||
| metadata: | ||
| name: ci-default | ||
| spec: | ||
| controlPlane: | ||
| ref: | ||
| apiVersion: controlplane.cluster.x-k8s.io/v1beta1 | ||
| kind: KubeadmControlPlaneTemplate | ||
| name: ci-default-kubeadm-control-plane | ||
| machineInfrastructure: | ||
| ref: | ||
| kind: AzureMachineTemplate | ||
| apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 | ||
| name: ci-default-control-plane | ||
| infrastructure: | ||
| ref: | ||
| apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 | ||
| kind: AzureClusterTemplate | ||
| name: ci-default-azure-cluster | ||
| workers: | ||
| machineDeployments: | ||
| - class: ci-default-worker | ||
| template: | ||
| bootstrap: | ||
| ref: | ||
| apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 | ||
| kind: KubeadmConfigTemplate | ||
| name: ci-default-worker | ||
| infrastructure: | ||
| ref: | ||
| apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 | ||
| kind: AzureMachineTemplate | ||
| name: ci-default-worker | ||
| machineHealthCheck: | ||
| maxUnhealthy: 100% | ||
| unhealthyConditions: | ||
| - type: E2ENodeUnhealthy | ||
| status: "True" | ||
| timeout: 30s | ||
| - class: ci-default-worker-win | ||
| template: | ||
| bootstrap: | ||
| ref: | ||
| apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 | ||
| kind: KubeadmConfigTemplate | ||
| name: ci-default-worker-win | ||
| infrastructure: | ||
| ref: | ||
| apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 | ||
| kind: AzureMachineTemplate | ||
| name: ci-default-worker-win | ||
| machineHealthCheck: | ||
| maxUnhealthy: 100% | ||
| unhealthyConditions: | ||
| - type: E2ENodeUnhealthy | ||
| status: "True" | ||
| timeout: 30s | ||
| --- | ||
| apiVersion: controlplane.cluster.x-k8s.io/v1beta1 | ||
| kind: KubeadmControlPlaneTemplate | ||
| metadata: | ||
| name: ci-default-kubeadm-control-plane | ||
| spec: | ||
| template: | ||
| spec: | ||
| kubeadmConfigSpec: | ||
| clusterConfiguration: | ||
| apiServer: | ||
| extraArgs: | ||
| cloud-config: /etc/kubernetes/azure.json | ||
| cloud-provider: azure | ||
| extraVolumes: | ||
| - hostPath: /etc/kubernetes/azure.json | ||
| mountPath: /etc/kubernetes/azure.json | ||
| name: cloud-config | ||
| readOnly: true | ||
| timeoutForControlPlane: 20m | ||
| controllerManager: | ||
| extraArgs: | ||
| allocate-node-cidrs: "false" | ||
| cloud-config: /etc/kubernetes/azure.json | ||
| cloud-provider: azure | ||
| extraVolumes: | ||
| - hostPath: /etc/kubernetes/azure.json | ||
| mountPath: /etc/kubernetes/azure.json | ||
| name: cloud-config | ||
| readOnly: true | ||
| etcd: | ||
| local: | ||
| dataDir: /var/lib/etcddisk/etcd | ||
| extraArgs: | ||
| quota-backend-bytes: "8589934592" | ||
| diskSetup: | ||
| filesystems: | ||
| - device: /dev/disk/azure/scsi1/lun0 | ||
| extraOpts: | ||
| - -E | ||
| - lazy_itable_init=1,lazy_journal_init=1 | ||
| filesystem: ext4 | ||
| label: etcd_disk | ||
| - device: ephemeral0.1 | ||
| filesystem: ext4 | ||
| label: ephemeral0 | ||
| replaceFS: ntfs | ||
| partitions: | ||
| - device: /dev/disk/azure/scsi1/lun0 | ||
| layout: true | ||
| overwrite: false | ||
| tableType: gpt | ||
| files: | ||
| - contentFrom: | ||
| secret: | ||
| key: control-plane-azure.json | ||
| name: replace_me | ||
| owner: root:root | ||
| path: /etc/kubernetes/azure.json | ||
| permissions: "0644" | ||
| initConfiguration: | ||
| nodeRegistration: | ||
| kubeletExtraArgs: | ||
| azure-container-registry-config: /etc/kubernetes/azure.json | ||
| cloud-config: /etc/kubernetes/azure.json | ||
| cloud-provider: azure | ||
| name: '{{ ds.meta_data["local_hostname"] }}' | ||
| joinConfiguration: | ||
| nodeRegistration: | ||
| kubeletExtraArgs: | ||
| azure-container-registry-config: /etc/kubernetes/azure.json | ||
| cloud-config: /etc/kubernetes/azure.json | ||
| cloud-provider: azure | ||
| name: '{{ ds.meta_data["local_hostname"] }}' | ||
| mounts: | ||
| - - LABEL=etcd_disk | ||
| - /var/lib/etcddisk | ||
| postKubeadmCommands: [ ] | ||
| preKubeadmCommands: [ ] | ||
| --- | ||
| apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 | ||
| kind: AzureClusterTemplate | ||
| metadata: | ||
| name: ci-default-azure-cluster | ||
| spec: | ||
| template: | ||
| spec: | ||
| additionalTags: | ||
| replace_me_key: replace_me_val | ||
| identityRef: | ||
| apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 | ||
| kind: AzureClusterIdentity | ||
| location: replace_me | ||
| networkSpec: | ||
| subnets: | ||
| - role: control-plane | ||
| - natGateway: | ||
| name: node-natgateway | ||
| role: node | ||
| --- | ||
| apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 | ||
| kind: AzureMachineTemplate | ||
| metadata: | ||
| name: ci-default-control-plane | ||
| spec: | ||
| template: | ||
| spec: | ||
| dataDisks: | ||
| - diskSizeGB: 256 | ||
| lun: 0 | ||
| nameSuffix: etcddisk | ||
| osDisk: | ||
| diskSizeGB: 128 | ||
| osType: Linux | ||
| sshPublicKey: "" | ||
| vmSize: replace_me | ||
| --- | ||
| apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 | ||
| kind: KubeadmConfigTemplate | ||
| metadata: | ||
| name: ci-default-worker | ||
| spec: | ||
| template: | ||
| spec: | ||
| files: | ||
| - contentFrom: | ||
| secret: | ||
| key: worker-node-azure.json | ||
| name: replace_me | ||
| owner: root:root | ||
| path: /etc/kubernetes/azure.json | ||
| permissions: "0644" | ||
| joinConfiguration: | ||
| nodeRegistration: | ||
| kubeletExtraArgs: | ||
| azure-container-registry-config: /etc/kubernetes/azure.json | ||
| cloud-config: /etc/kubernetes/azure.json | ||
| cloud-provider: azure | ||
| name: '{{ ds.meta_data["local_hostname"] }}' | ||
| preKubeadmCommands: [ ] | ||
| --- | ||
| apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 | ||
| kind: AzureMachineTemplate | ||
| metadata: | ||
| name: ci-default-worker | ||
| spec: | ||
| template: | ||
| spec: | ||
| osDisk: | ||
| diskSizeGB: 128 | ||
| osType: Linux | ||
| sshPublicKey: "" | ||
| vmSize: replace_me | ||
| --- | ||
| apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 | ||
| kind: KubeadmConfigTemplate | ||
| metadata: | ||
| name: ci-default-worker-win | ||
| spec: | ||
| template: | ||
| spec: | ||
| preKubeadmCommands: [] | ||
| postKubeadmCommands: | ||
| - nssm set kubelet start SERVICE_AUTO_START | ||
| - powershell C:/defender-exclude-calico.ps1 | ||
| joinConfiguration: | ||
| nodeRegistration: | ||
| name: '{{ ds.meta_data["local_hostname"] }}' | ||
| criSocket: npipe:////./pipe/containerd-containerd | ||
| kubeletExtraArgs: | ||
| cloud-provider: azure | ||
| cloud-config: 'c:/k/azure.json' | ||
| azure-container-registry-config: 'c:/k/azure.json' | ||
| feature-gates: "WindowsHostProcessContainers=true" | ||
| v: "2" | ||
| windows-priorityclass: "ABOVE_NORMAL_PRIORITY_CLASS" | ||
| files: | ||
| - contentFrom: | ||
| secret: | ||
| key: worker-node-azure.json | ||
| name: replace_me | ||
| owner: root:root | ||
| path: c:/k/azure.json | ||
| permissions: "0644" | ||
| - path: C:/defender-exclude-calico.ps1 | ||
| permissions: "0744" | ||
| content: |- | ||
| Add-MpPreference -ExclusionProcess C:/opt/cni/bin/calico.exe | ||
| Add-MpPreference -ExclusionProcess C:/opt/cni/bin/calico-ipam.exe | ||
| --- | ||
| apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 | ||
| kind: AzureMachineTemplate | ||
| metadata: | ||
| name: ci-default-worker-win | ||
| annotations: | ||
| runtime: containerd | ||
| spec: | ||
| template: | ||
| spec: | ||
| osDisk: | ||
| osType: "Windows" | ||
| diskSizeGB: 128 | ||
| managedDisk: | ||
| storageAccountType: "Premium_LRS" | ||
| sshPublicKey: "" | ||
| vmSize: replace_me | ||
9 changes: 9 additions & 0 deletions
9
templates/test/ci/prow-clusterclass-ci-default/kustomization.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| apiVersion: kustomize.config.k8s.io/v1beta1 | ||
| kind: Kustomization | ||
| namespace: default | ||
| resources: | ||
| - base.yaml | ||
| - ../../../azure-cluster-identity | ||
| patchesStrategicMerge: | ||
| - patches.yaml | ||
| - variables.yaml |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just curious, is "replace_me" an arbitrary string or is it some agreed upon contract/pattern for required variables in clusterclass?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's unfortunately just a workaround for fields that have to be set for the validating webhook and are then later overwritten per cluster.
I think this workaround kind of becomes a pattern right now, but there is no standardization (yet).
Not sure how we can improve this, maybe it's an option that these fields should be optional when the resource is referenced in a ClusterClass. But not sure how feasible this is.
(as far as I'm aware CAPA is using a similar but not the same string)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CAPA is using REPLACEME. I don't particularly have a preference but I vaguely remember using
replace_mebeing used in other places where a similar mechanism was involved.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm inferring from reading this history that empty string val doesn't suffice?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Afaik the validation webhook will complain
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a capi issue tracking how we might improve this? IMO this is a fairly significant bit of UX friction for users to deal with, we should do this work on behalf of them (happy to help).
Everything else here actually looks fine to me and test signal is green, I'm going to mark this as /lgtm.
Thanks so much @shysank, really happy to have ClusterClass coverage in capz!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure, we probably have some guidance around cluster-specific fields and that they shouldn't be added to cluster templates (which was in general nicely done by CAPZ). But I"m not sure if that falls under the "cluster-specific" field category.
I think it makes sense to open an issue in CAPI and see what we have / need.
We can discuss documentation improvements or potential implementation changes to mitigate this issue (e.g. maybe some fields shouldn't be mandatory on a cluster template / machine template when it's used as template in a ClusterClass).