Tracking issue for clusterctl v2 implementation

[fabriziopandini]

Tracking issue for clusterctl v2 work

Initial work based on the clusterctl POC

POC code walkthrough: 1 and 2

IMPORTANT! Details about the following activity breakdown are available https://docs.google.com/document/d/1rLu1P7Bs6BJ1ZL6aOz3AAWGwQ9N8Q3hle7PC56G11t0/edit?usp=sharing

• [Fabrizio] ⚠️ Clusterctlv2 cli scaffolding #1730 Delete current clusterctl and replace with a new CLI stub; finalize the UX
• [Fabrizio] ⚠️ Clusterctlv2 api #1738 Add CRD for clusterctl metadata about installed providers; finalize the API
• [Fabrizio] ⚠️ Clusterctlv2 low-level config library #1762 Add the low-level library for managing clusterctl configurations with default backend implementation based on viper
  • [Fabrizio] ✨clusterctl: rename config file #2040 rename config
• [Fabrizio] ⚠️ clusterctlv2 add implementation for clusterctl config providers #1864 implement high-level library and CLI for clusterctl configurations
• [Fabrizio] ⚠️ Clusterctlv2 add interfaces for the low-level Repository library #1842 Add the low-level library for managing provider's repository
  • [Fabrizio] ✨clusterctl: add support for local override of files read from provider's repository #1981 add support for local overrides
• [Jiatong] ⚠️ clusterctl v2 add the low-level library for managing provider's repository using Github #1909 Implement first backend implementation for repository using Github
• [Fabrizio] ⚠️ clusterctlv2 add low-level library for the provider components YAML #1891 Add the low-level library for managing the component.yaml file read from provider's repositories
• [Arvinderpal] ⚠️ clusterctlv2: add file system backend implementations for provider's repository #1963 Add file system backend implementations for provider's repository
• [low priority] Add more backend implementations for provider's repository backends (HTTP/HTTPS, S3?)
• [Fabrizio] ⚠️ clusterctlv2: add interfaces for the management cluster low level library #1898 Add the low-level library for management clusters / handling of providers inventory
  • [Fabrizio] kubernetes-sigs/cluster-api: Pull Request 1986 fix RBAC for multienancy
• [Fabrizio] ⚠️ clusterctlv2: add the clusterctl init command #1932 Add the high-level library and CLI for init
  • [Fabrizio] 🐛clusterctl: fix-init #2008 fix for init
  • [Fabrizio] ⚠️ clusterctl: installs cert-manager during init #2017 install cert-manager
  • [Fabrizio] ✨clusterctl: implements hack for generating local overrides #2027 hack for local testing
  • [Fabrizio] ✨ add clusterctl-settings.json cluster-api-provider-aws#1470 cfg for the hack
  • [Fabrizio] add clusterctl-settings.json cluster-api-provider-vsphere#716 cfg for the hack
  • [Fabrizio] 🐛fix clusterctl hack for python 3.0 #2075 fix for python 3
  • [Fabrizio] 🐛clusterctl: remove --force flag from init #2071 remove --force
  • [Fabrizio] 🐛clusterctl: fix docker provider #2098 fix the docker provider
  • [Fabrizio] ✨clusterctl: add support for the controlplane provider #2099 support for the control plane provider
  • [Fabrizio] 🐛clusterctl: wait inventory crd #2206 wait for inventory CRD
• [Fabrizio] ✨clusterctl: implement cluster templates-low-level-lib #1996 Add the low-level library for managing cluster templates read from provider's repositories
  • [Fabrizio] ⚠️ clusterctl: allow config variable override #1975 Allow variables override
  • [Fabrizio] ✨clusterctl: small refactor of fixNamespace #1976 Refactor fix namespace
• [Fabrizio] ⚠️ clusterctl: add the clusterctl config cluster command #2047 Add the high-level library and CLI for cluster templates
  NB. To be changed vs the POC, rif ⚠️ Clusterctlv2 cli scaffolding #1730 (comment)
  • [Fabrizio] 🐛clusterctl: fix config provider command #2211 fix config provider command
  • [Fabrizio] 🐛clusterctl: simplify cluster template naming #2221 Simplify naming
  • [Fabrizio] ✨clusterctl: read templates from different sources #2265 Allow cluster templates from user defined repository
• [Fabrizio] ✨clusterctl: add the config provider <provider-name> command #2048 Add the low-level library he high-level library and CLI for config provider
• [Fabrizio] Add the low-level library for Move
  NB. To be changed vs the POC, rif Define clusterctl move process #1525
  • [Fabrizio] ✨clusterctl: add test framework for move #1977 test framework
  • [Fabrizio] ✨clusterctl: improve move test framework #2059 improve test framework
  • [Fabrizio] ✨clusterctl: create the object graph for move #2060 object graph
  • [Fabrizio] ✨Clusterctl: Add the ObjectMover service #2102 object mover
  • [Fabrizio] ✨clusterctl: add the clusterctl move command #2130 Move CLI
  • [Fabrizio] ✨clusterctl: move shared objects #2161 Move shared objects
  • [Fabrizio] 🐛clusterctl: move ensure all the namespaces exists #2245 Ensure NS
  • [Fabrizio] ✨clusterctl: add a preflight check for provisioning completed before move #2246 Preflight provisioning complete
• [Fabrizio] ✨clusterctl: add the clusterctl move command #2130 Add the high-level library and CLI for Move
  NB. To be changed vs the POC, see ⚠️ Clusterctlv2 cli scaffolding #1730 (comment), ⚠️ Clusterctlv2 cli scaffolding #1730 (comment)
• [Fabrizio] ✨clusterctl: add the delete command #2078 Add the high-level library and CLI for delete
• [Fabrizio] ✨clusterctl: add version command #2061 Version command (not included in the POC)
• [Fabrizio] Upgrade command (not included in the POC)
  • [Fabrizio] ⚠️ clusterctl: add metadata #1974 add metadata
  • [Fabrizio] ✨clusterctl: add Repository.GetVersions method #1978 add repository.GetVersions
  • [Fabrizio] ✨clusterctl: add upgrader with the upgrade plan logic #2134 upgrade plan
  • [Fabrizio] ✨clusterctl: add upgrade plan cmd #2178 upgrade plan cmd
  • [Fabrizio] ✨clusterctl: implement upgrade apply #2281 upgrade apply cmd
• Pre pull images
  • [Fabrizio] ✨clusterctl: inspect images in provider components #2104 inspect images
• [Fabrizio] Adopt providers
• [Arvinderpal] ✨ clusterctl e2e tests #2236 E2E tests (not included in the POC)
• [Fabrizio] 📖 clusterctl: initial set of documentation #1994 Review clusterctl docs in the book (not included in the POC)
  • ✨clusterctl: add godoc and doc #2175 More docs
• [Fabrizio] Logging
  • [Fabrizio] ✨clusterctl: add logger #2150 Add logger
  • CLOSED [Fabrizio] ✨clusterctl: log propagation #2151 Log Propagation
  • [Fabrizio] ✨clusterctl use global logger #2191 Use global logger
• Miscellanea
  • [Fabrizio] ✨clusterctl: fix labels #2152 fix labels
  • [Fabrizio] 🐛clusterctl: fix public library inconsistencies #2220 refactor libray

Potential code improvements/cleanups

• ⚠️ Clusterctlv2 add interfaces for the low-level Repository library #1842 (review) (library layout)
• ⚠️ clusterctlv2: add interfaces for the management cluster low level library #1898 (comment) (more tests)
• ⚠️ clusterctlv2: add the clusterctl init command #1932 (comment) (audit on API names)
• ⚠️ clusterctlv2: add the clusterctl init command #1932 (comment) (rollback on init)
• ⚠️ clusterctlv2: add the clusterctl init command #1932 (comment) (ensure type imple interface in test --> no consensus yet)
• Done - ⚠️ clusterctlv2: add the clusterctl init command #1932 (comment) (cleanup init retrun params)
• ⚠️ clusterctlv2: add the clusterctl init command #1932 (comment) (cleanup tests)
• Done - ⚠️ clusterctlv2: add the clusterctl init command #1932 (comment) (logs)
• Done - ✨Clusterctl: Add the ObjectMover service #2102 (comment) (logs)
• Done - ✨clusterctl: add the delete command #2078 (comment) (labels/providers names)
• Done - ✨clusterctl: add the delete command #2078 (comment) (labels)

Future work

• Prototype interactive mode for allowing users to set "on-the-fly" the variables to be injected in the yaml for providers components or providers templates
• Prototype a pluggable template system (vs supporting only variables substitution)

[vincepri]

/milestone v0.3.0
/area clusterctl
/priority important-soon

[ncdc]

/assign @fabriziopandini
for overall ownership 😄

[fabriziopandini]

We are done for the upcoming RC, except eventually last minutes changes for the webhooks

/close
🎉 🎉 🎉 🎉 🎉

#2313, #2314 are small fixes, not blocking for the release.

Clusterctl adopts, E2E tests, and other minor improvements code/cleanup are shifting to the next round.

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

We are done for the upcoming RC, except eventually last minutes changes for the webhooks

/close
🎉 🎉 🎉 🎉 🎉

#2313, #2314 are small fixes, not blocking for the release.

Clusterctl adopts, E2E tests, and other minor improvements code/cleanup are shifting to the next round.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.