[epic] Initial kubectl operator plugin updates for operator-controller and catalogd

[joelanford]

With the imminent 1.0.0 releases of operator-controller and catalogd, we should circle back to our kubectl-operator plugin, update it to work with operator-controller/catalogd APIs and make those interactions the default experience (deprecating the OLMv0 implementation).

One of the most important new features the kubectl plugin should include is a way to query the contents of the ClusterCatalog objects on a cluster. This is a tricky problem to solve because the networking and authentication stacks between an off-cluster client and the on-cluster catalogd service are often environment-specific. Potential options include:

• kubectl port-forward, but this requires users to have permission to get information about the catalogd pod (and possibly its service) in order to setup the port-forward connection.
• kubectl proxy, but no client authentication to the in-cluster service is supported. This would be a problem if we ever want to require client authentication in order to access catalogd from outside the cluster. Are there any advantages of proxy over port-forward that we should consider?
• An Ingress or Gateway - these APIs are the traditional way to expose cluster services outside the cluster, but there are many implementations, each with nuances, and OLM cannot really make assumptions that there will even be an ingress or gateway controller running on every cluster.
• Service with a NodePort - this configures each node to start a listener on the specified port for the specified protocol, and all nodes proxy connections to the in-cluster service. Downside is that it requires a port reservation, and clients would need a way to discover the node IPs and port assignment (not insurmountable, but something we'd have to consider carefully).

Because all of these options have pitfalls, we may need to design a flexible solution that enables distributions that include OLM to select which of these to support. And ideally the client could automatically discover which technique to use. Since clients will be required to query for ClusterCatalog to know what is even available to query, perhaps catalogd could include external URLs in the ClusterCatalog status?

---

In addition to the challenges with exposing catalog content off-cluster, we should also implement some or all of:

• ClusterExtension get/list/install/upgrade/uninstall
• ClusterCatalog get/list/add/remove, and maybe mechanisms to update availability, labels, priority, etc.

We should definitely plan to implement a really nice UX for discovering the current state of what is actually present on the cluster. We can implement custom printer columns, sorting, etc. that can really enhance a user's understanding of the state of the system.

---

There may be additional iterations on this effort (for e.g. as we discover possible benefits of new printer columns or something else), but for this initial iteration we plan a functional breakdown to give solid experiences in stages. This may enable other features (for e.g. service account permissions evaluations).

These phases are captured as sub-issues, and duplicated here for those expecting a table:

• Extend kubectl-operator to get/list OLMv1 resources #1767
• Extend kubectl-operator to list available OLMv1 bundles for install. #1768
• Extend kubectl-operator to install and uninstall ClusterExtensions #1769
• Extend kubectl-operator to add/remove ClusterCatalogs #1770
• Extend kubectl-operator to modify existing ClusterCatalogs #1771
• Extend kubectl-operator plugin to upgrade a ClusterExtension #1772
• Spike: Expose catalogd service off cluster #1765
• Investigate listing cluster-catalog contents of very large catalogs with port-forwarding #1853
• Deprecate olm v0 components within kubectl-operator plugin #1766
• Prepare and include demo of olmv1 commands in kubectl-operator plugin README #1807
• Move the olmv1 implementation in kubectl-operator to the operator-controller repo #1850
• Optimize olmv1 kubectl plugin UX #1851
• Publish olmv1 kubectl plugin #1852

[LalatenduMohanty]

@joelanford have the POC PR operator-framework/kubectl-operator#213