Revert proposal change of ClusterStatus.APIEndpoints to APIEndpoint

[moshloop]

APIEndpoint is only relevant in environments with native load balancers, For on-premise APIEndpoints is the logical field for client-side load balancers to use.

[detiber]

@moshloop I'm not sure I agree. In general this has been used centrally for determining how to contact the API server for a cluster, which for the purpose of Kubernetes clients is required to be a single endpoint. If we need to track multiple ControlPlane Endpoints, we should probably track those separately rather than overload a field intended to be used by clients.

[detiber]

/assign @ncdc

[moshloop]

It doesn't need to be a single endpoint, just a single endpoint exposed to clients.

Currently I have a client-side load balancer using haproxy + consul, and all clients (kubelet, kubeadm, etcc) connect to haproxy on localhost:8443 as the single endpoint.

I was planning on replacing the consul component with APIEndpoints,

[vincepri]

How are you using multiple endpoints in a kubeconfig or in client-go?

[detiber]

@moshloop how would an external management server know where to contact the Cluster's API Server in order to check Node status for the NodeRef controller in that model? Or how would a workload cluster administrator determine how to interact with their Cluster?

[moshloop]

How are you using multiple endpoints in a kubeconfig or in client-go?

With a client-side loadbalancer deployed out-of-band from k8s e.g. localhost:8443

[moshloop]

how would an external management server know where to contact the Cluster's API Server

Pick any random APIEndpoint from the list (performing a type of client-side load balancing)

ow would a workload cluster administrator determine how to interact with their Cluster?

Pick any random APIEndpoint from the list or rely on something like DNS round-robin

[ncdc]

Are there any v1alpha1 Cluster API providers that set more than 1 entry in the APIEndpoints array today?

[detiber]

Are there any v1alpha1 Cluster API providers that set more than 1 entry in the APIEndpoints array today?

I know the vSphere provider does this today.

[ncdc]

Given that we originally said we didn't want to make any data model or behavioral changes to the Cluster other than removing the actuators, I think it's fair to merge this and we'll file a separate issue to discuss apiserver endpoint(s) and how internal CAPI consumers use them vs what an external client should use.

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: moshloop, ncdc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ncdc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[moshloop]

From what I can see CAPV sets the API Endpoint to the endpoint used in the saved kubeconfig:

https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/master/pkg/cloud/vsphere/actuators/cluster/actuator.go#L196

This is flawed and something I want to fix, the flow should be:

1. Pick a random endpoint to create a remote client (using saved or dynamically generated certs)
   1a) Requeue if the endpoint is not working
2. Update APIEndpoints with the list of active ready masters.

@akutz / @timothysc any thoughts on this?

[detiber]

Pick a random endpoint to create a remote client (using saved or dynamically generated certs)
1a) Requeue if the endpoint is not working

This isn't default behavior for kubernetes clients, which expect a single stable endpoint.

[akutz]

CAPV doesn’t set APIEndpoints to more than one value, and Moshe’s evaluation isn’t completely accurate either.

In other words, CAPV uses a single endpoint for APIEndpoints.

If there’s HA then the load balancer FQDN is used, otherwise the API endpoint of the oldest control plane node is used.

[akutz]

Seems odd to merge this when there were outstanding questions on CAPV for which no authoritative answer was provided. I was pinged to answer a question, but apparently there was a time limit to respond of which I was unaware.

[ncdc]

@akutz this reverts a data model change (in the proposal, at least) so it is back in line with v1alpha1 pending further discussion in #1197

[akutz]

I liked the change and was I favor of it, and didn’t think CAPV should be used to support the revert, that’s all.

[texascloud]

@moshloop I don't understand why your client-side load balancing solution requires this change. It sounds like it's working for you