Clusterctl v2: support for additional backend implementations for provider repository

[Arvinderpal]

User Story

As an operator, I would like to customize my CAPI core and provider specific data and have clusterctl pull that data from a backend other than github.

As a developer, I would like to test my releases without having to publish them to github.

Detailed Description

Clusterctl v2 will support fetching CAPI core and provider specific data (i.e. component yamls, templates) directly from github repositories that use the release feature to publish artifacts and versions. We should add support for backends beyond github, such as local file system, S3, GCE Buckets, etc.

Anything else you would like to add:

Here are a few ways to going about this:

1. clusterctl currently defines a type Repository interface. Each backend implementation could implement this interface. This is already being done for github (⚠️ clusterctl v2 add the low-level library for managing provider's repository using Github #1909) and local file system (⚠️ clusterctlv2: add file system backend implementations for provider's repository #1963). The same approach could be applied to S3, GCE, etc.
2. We could utilize an existing library, for example, go-getter, that allows you to fetch data from various backends.
3. We could leave it to the user to pre-fetch data. Similarly, high-level tools that consume clusterctl as a library would be responsible for pre-fetching the data. From clusterctl's perspective, the data only need reside on the local file system and adhere to the standard directory structure defined in the local file system backend implementation.

First two options are more convinent for the user, but will create addtional dependencies in clusterctl; for example, aws-sdk-go/aws for S3 and cloud.google.com/go/storage for GCE. Option 2, with go-getter, may require some tweaks to current type Repository interface model but has the benefit of removing backend specific code from clusterctl.

/kind feature

[Arvinderpal]

@fabriziopandini

[fabriziopandini]

1. clusterctl currently defines a type Repository interface. Each backend implementation could implement this interface.

I think the idea of a pluggable backend should be preserved (and eventually improved)

@Arvinderpal please be aware that currently, the work on new backends is on hold on the clusterctl tracking issue because most probably there will be some rework in the Repository interface, due to new requirements under definition for supporting upgrades.

2. We could utilize an existing library, for example, go-getter, that allows you to fetch data from various backends.

I defer to others the decision about importing a new dependency, but I'm generally in favor of supporting more backends for clusterctl if they fit into the Repository interface

3. We could leave it to the user to pre-fetch data.

I'm ok with this if this is a special implementation of a pluggable backend; I have also in mind a possible solution for the developer use case (allow local override of repository files without requiring a "full fledged repository implementation"), but before I'm waiting to complete the design on upgrades

[k8s-ci-robot]

@fabriziopandini: The label(s) area/clustrctl cannot be applied, because the repository doesn't have them

In response to this:

/area clustrctl

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@fabriziopandini: The label(s) area/clustrctl cannot be applied, because the repository doesn't have them

In response to this:

/area clustrctl

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fabriziopandini]

/area clusterctl

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[vincepri]

/lifecycle frozen

[fabriziopandini]

@vincepri what about closing this issue unless a more specific use case arises?

[vincepri]

SGTM

/close

[k8s-ci-robot]

@vincepri: Closing this issue.

In response to this:

SGTM

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.