deploy CSI with ClusterResourceSet

[akutz]

/kind feature

Describe the solution you'd like
This is to discuss whether or not CSI should be installed as an add-on, and not reconciled as part of CAPV when deploying workload clusters.

cc @voor @mylesagray

[mylesagray]

@akutz is there more context for this? A meeting recording or such that I missed? Is the proposal to remove CSI as part of the default installation when using CAPV?

[akutz]

Per my discussion with @mylesagray we decided a good cutline for what is an external add-on should be whether the platform leveraged by the "add-on" is built into vSphere. Since CSI depends on container network storage (CNS), and it is built into vSphere, CSI can be deployed by CAPV.

On the other hand, the NSX container plug-in (NCP), a container network interface (CNI) solution, should be deployed as an add-on since NSX is not built into vSphere.

[akutz]

I will leave this issue open until someone files a PR that adds documentation to the effect of the above.

[akutz]

/help

[k8s-ci-robot]

@akutz:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yastij]

@akutz - I think we should re-visit this assumption, as any fix to CSI implies cutting a new release of CAPV.

[voor]

Since everything is manifest based (i.e., you're not putting static pods in anywhere) this also makes a lot more sense now, especially with the post-cluster apply stuff getting more fleshed out.

Referencing kubernetes-sigs/cluster-api#2395

[MaxRink]

We honestly would also prefer it to be an add-on, as we have instances where we cant run the csi provider but would still like to provision clusters via CAPV

[yastij]

@MaxRink - this is definitely happening