clusterctl create fails with "TLS handshake timeout" error

[yukirii]

/kind bug

What steps did you take and what happened:

Tried to create new cluster with clusterctl create cluster. After kube-apiserver starts up, the command fails with "TLS handshake timeout" error:

clusterctl create cluster -v10 --minikube kubernetes-version=v1.12.3 --provider openstack -c centos/out/cluster.yaml -m centos/out/machines.yaml -p centos/out/provider-components.yaml

I0106 23:55:49.830255   15761 clusterdeployer.go:283] Waiting for kubeconfig on cluster-api-test-master-w4zdg to become ready...
I0106 23:55:49.830318   15761 deployer.go:51] Returning IP from machine annotation <master IP address>
I0106 23:55:50.181421   15761 loader.go:359] Config loaded from file /var/folders/1p/s5033srj1pb0jp8bmk_5_jh46v4m0c/T/830609860
I0106 23:55:50.182020   15761 clusterdeployer.go:132] Applying Cluster API stack to target cluster
I0106 23:55:50.182031   15761 clusterdeployer.go:260] Applying Cluster API Provider Components
I0106 23:55:50.182037   15761 clusterclient.go:521] Waiting for kubectl apply...
I0106 23:55:50.412771   15761 clusterclient.go:526] Waiting for kubectl apply... server not yet available: couldn't kubectl apply: exit status 1, output: The connection to the server <master IP address>:6443 was refused - did you specify the right host or port?
I0106 23:56:00.416335   15761 clusterclient.go:521] Waiting for kubectl apply...
I0106 23:56:00.506340   15761 clusterclient.go:526] Waiting for kubectl apply... server not yet available: couldn't kubectl apply: exit status 1, output: The connection to the server <master IP address>:6443 was refused - did you specify the right host or port?
I0106 23:56:10.415464   15761 clusterclient.go:521] Waiting for kubectl apply...
W0106 23:56:20.510616   15761 clusterclient.go:537] Waiting for kubectl apply... unknown error couldn't kubectl apply: exit status 1, output: Unable to connect to the server: net/http: TLS handshake timeout
I0106 23:56:20.511589   15761 createbootstrapcluster.go:37] Cleaning up bootstrap cluster.
I0106 23:56:20.511635   15761 minikube.go:58] Running: minikube [delete]
I0106 23:56:21.574570   15761 minikube.go:62] Ran: minikube [delete] Output: Deleting local Kubernetes cluster...
Machine deleted.
F0106 23:56:21.575171   15761 create_cluster.go:64] unable to apply cluster api stack to target cluster: unable to apply cluster api controllers: couldn't kubectl apply: exit status 1, output: Unable to connect to the server: net/http: TLS handshake timeout

What did you expect to happen:

The target cluster should be created and clusterctl shouldn't fail.

Anything else you would like to add:

I tried kubectl apply manually just after kube-apiserver started. I could reproduce similar errors:

% kubectl -v 6 --kubeconfig ./kubeconfig apply -f test.yml
I0106 22:08:41.129297     559 loader.go:357] Config loaded from file ./kubeconfig
I0106 22:08:41.185384     559 round_trippers.go:405] GET https://<master IP address>:6443/openapi/v2  in 52 milliseconds
I0106 22:08:41.185501     559 helpers.go:219] Connection error: Get https://<master IP address>:6443/openapi/v2: dial tcp <master IP address>:6443: getsockopt: connection refused
F0106 22:08:41.185533     559 helpers.go:119] The connection to the server <master IP address>:6443 was refused - did you specify the right host or port?

% kubectl -v 6 --kubeconfig ./kubeconfig apply -f test.yml
I0106 22:08:44.374345     615 loader.go:357] Config loaded from file ./kubeconfig
I0106 22:08:54.393620     615 round_trippers.go:405] GET https://<master IP address>:6443/openapi/v2  in 10015 milliseconds
I0106 22:08:54.399416     615 helpers.go:219] Connection error: Get https://<master IP address>:6443/openapi/v2: net/http: TLS handshake timeout
F0106 22:08:54.399455     615 helpers.go:119] Unable to connect to the server: net/http: TLS handshake timeout

After that, kubectl apply command worked fine without errors. It looks like these errors are occurred temporarily if kube-apiserver is starting up.

I'm sure this error occured in

cluster-api/cmd/clusterctl/clusterdeployer/clusterclient/clusterclient.go

Line 537 in f03f47c

klog.Warningf("Waiting for kubectl apply... unknown error %v", err)

It looks like the handshake error is processed as "unknown error", and clusterctl fails without retrying.

Environment:

• Cluster-api version:

   revision = "3547f8dd93078c0b0f0b221ee1c2a57f96673081"
   version = "0.0.0-alpha.4"
  

  • using with cluster-api-provider-openstack (revision: f1dae384e359f6f7e266089d10609f9e1a55aaa5)
• Kubernetes version: (use kubectl version):

   Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
   Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
  

• OS (e.g. from /etc/os-release):
  CentOS Linux release 7.2.1511 (Core)

[roberthbailey]

This seems like a provider specific issue. Can you please open an issue in https://github.com/kubernetes-sigs/cluster-api-provider-openstack instead of here? Thanks.