Kubernetes-anyhwere: make destroy is flaky

[abrarshivani]

Kubernetes-anywhere issue linked here.

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): 1.4.4

Environment:

• Cloud provider or hardware configuration: vSphere
• OS (e.g. from /etc/os-release): Photon OS

What happened:
make destroy

What you expected to happen:
VM's should be destroyed and terraform state should be cleaned up.

Anything else do we need to know:
Output when I run make destroy:

panic: runtime error: index out of range
2016/11/22 00:15:00 [DEBUG] plugin: terraform:
2016/11/22 00:15:00 [DEBUG] plugin: terraform: goroutine 8 [running]:
2016/11/22 00:15:00 [DEBUG] plugin: terraform: panic(0x27f4e60, 0xc42000e0f0)
2016/11/22 00:15:00 [DEBUG] plugin: terraform: 	/opt/go/src/runtime/panic.go:500 +0x1a1
2016/11/22 00:15:00 [DEBUG] plugin: terraform: github.com/hashicorp/terraform/builtin/providers/vsphere.resourceVSphereVirtualMachineRead(0xc42033fe00, 0x2d2bb40, 0xc4203e3030, 0x1, 0x17)
2016/11/22 00:15:00 [DEBUG] plugin: terraform: 	/opt/gopath/src/github.com/hashicorp/terraform/builtin/providers/vsphere/resource_vsphere_virtual_machine.go:1060 +0x3458
2016/11/22 00:15:00 [DEBUG] plugin: terraform: github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc4203de0c0, 0xc42021b300, 0x2d2bb40, 0xc4203e3030, 0xc42030b280, 0x1, 0x18)
2016/11/22 00:15:00 [DEBUG] plugin: terraform: 	/opt/gopath/src/github.com/hashicorp/terraform/helper/schema/resource.go:259 +0x131
2016/11/22 00:15:00 [DEBUG] plugin: terraform: github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc4203da570, 0xc42021b2c0, 0xc42021b300, 0x0, 0x18, 0x18)
2016/11/22 00:15:00 [DEBUG] plugin: terraform: 	/opt/gopath/src/github.com/hashicorp/terraform/helper/schema/provider.go:201 +0x91
2016/11/22 00:15:00 [DEBUG] plugin: terraform: github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc42030fbc0, 0xc4203d9100, 0xc4203d9730, 0x0, 0x0)
2016/11/22 00:15:00 [DEBUG] plugin: terraform: 	/opt/gopath/src/github.com/hashicorp/terraform/plugin/resource_provider.go:482 +0x4e
2016/11/22 00:15:00 [DEBUG] plugin: terraform: reflect.Value.call(0xc42033f440, 0xc4203c6150, 0x13, 0x2da4800, 0x4, 0xc42054deb0, 0x3, 0x3, 0xc42034e6b0, 0xc4201bd920, ...)
2016/11/22 00:15:00 [DEBUG] plugin: terraform: 	/opt/go/src/reflect/value.go:434 +0x5c8
2016/11/22 00:15:00 [DEBUG] plugin: terraform: reflect.Value.Call(0xc42033f440, 0xc4203c6150, 0x13, 0xc42054deb0, 0x3, 0x3, 0xc42034e6a8, 0xc4201bd800, 0xda2681)
2016/11/22 00:15:00 [DEBUG] plugin: terraform: 	/opt/go/src/reflect/value.go:302 +0xa4
2016/11/22 00:15:00 [DEBUG] plugin: terraform: net/rpc.(*service).call(0xc4203ad800, 0xc4203ad7c0, 0xc4203e2120, 0xc4203c8b00, 0xc4203ea3e0, 0x255f980, 0xc4203d9100, 0x16, 0x255f9c0, 0xc4203d9730, ...)
2016/11/22 00:15:00 [DEBUG] plugin: terraform: 	/opt/go/src/net/rpc/server.go:383 +0x148
2016/11/22 00:15:00 [DEBUG] plugin: terraform: created by net/rpc.(*Server).ServeCodec
2016/11/22 00:15:00 [DEBUG] plugin: terraform: 	/opt/go/src/net/rpc/server.go:477 +0x421
2016/11/22 00:15:00 [ERROR] root: eval: *terraform.EvalRefresh, err: vsphere_virtual_machine.kubevm2: unexpected EOF
2016/11/22 00:15:00 [ERROR] root: eval: *terraform.EvalSequence, err: vsphere_virtual_machine.kubevm2: unexpected EOF
2016/11/22 00:15:00 [ERROR] root: eval: *terraform.EvalOpFilter, err: vsphere_virtual_machine.kubevm2: unexpected EOF
2016/11/22 00:15:00 [ERROR] root: eval: *terraform.EvalSequence, err: vsphere_virtual_machine.kubevm2: unexpected EOF
2016/11/22 00:15:00 [ERROR] root: eval: *terraform.EvalRefresh, err: vsphere_virtual_machine.kubevm1: unexpected EOF
2016/11/22 00:15:00 [TRACE] [walkRefresh] Exiting eval tree: vsphere_virtual_machine.kubevm2
2016/11/22 00:15:00 [ERROR] root: eval: *terraform.EvalSequence, err: vsphere_virtual_machine.kubevm1: unexpected EOF
2016/11/22 00:15:00 [ERROR] root: eval: *terraform.EvalOpFilter, err: vsphere_virtual_machine.kubevm1: unexpected EOF
2016/11/22 00:15:00 [ERROR] root: eval: *terraform.EvalSequence, err: vsphere_virtual_machine.kubevm1: unexpected EOF
2016/11/22 00:15:00 [TRACE] [walkRefresh] Exiting eval tree: vsphere_virtual_machine.kubevm1
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_node, got dep: vsphere_virtual_machine.kubevm2
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_node, got dep: tls_private_key.k8s-test5-master
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_node, got dep: provider.template
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_node, got dep: tls_self_signed_cert.k8s-test5-root
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_node, got dep: vsphere_virtual_machine.kubevm1
2016/11/22 00:15:00 [DEBUG] vertex null_resource.master, got dep: vsphere_virtual_machine.kubevm1
2016/11/22 00:15:00 [DEBUG] vertex provider.vsphere (close), got dep: vsphere_virtual_machine.kubevm1
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_master, got dep: vsphere_virtual_machine.kubevm1
2016/11/22 00:15:00 [DEBUG] vertex provider.vsphere (close), got dep: vsphere_virtual_machine.kubevm2
2016/11/22 00:15:00 [DEBUG] vertex provider.vsphere (close), got dep: vsphere_folder.cluster_folder
2016/11/22 00:15:00 [DEBUG] vertex data.tls_cert_request.k8s-test5-master, got dep: vsphere_virtual_machine.kubevm1
2016/11/22 00:15:00 [DEBUG] vertex null_resource.master, got dep: vsphere_virtual_machine.kubevm2
2016/11/22 00:15:00 [DEBUG] vertex data.tls_cert_request.k8s-test5-master, got dep: tls_private_key.k8s-test5-master
2016/11/22 00:15:00 [DEBUG] vertex null_resource.node2, got dep: vsphere_virtual_machine.kubevm1
2016/11/22 00:15:00 [DEBUG] vertex null_resource.node2, got dep: vsphere_virtual_machine.kubevm2
2016/11/22 00:15:00 [DEBUG] vertex tls_locally_signed_cert.k8s-test5-master, got dep: data.tls_cert_request.k8s-test5-master
2016/11/22 00:15:00 [DEBUG] vertex tls_locally_signed_cert.k8s-test5-master, got dep: tls_self_signed_cert.k8s-test5-root
2016/11/22 00:15:00 [DEBUG] vertex tls_locally_signed_cert.k8s-test5-master, got dep: provider.tls
2016/11/22 00:15:00 [DEBUG] vertex provider.tls (close), got dep: tls_locally_signed_cert.k8s-test5-master
2016/11/22 00:15:00 [DEBUG] vertex provider.tls (close), got dep: data.tls_cert_request.k8s-test5-node
2016/11/22 00:15:00 [DEBUG] vertex provider.tls (close), got dep: tls_locally_signed_cert.k8s-test5-admin
2016/11/22 00:15:00 [DEBUG] vertex provider.tls (close), got dep: tls_private_key.k8s-test5-master
2016/11/22 00:15:00 [DEBUG] vertex provider.tls (close), got dep: data.tls_cert_request.k8s-test5-master
2016/11/22 00:15:00 [DEBUG] vertex provider.tls (close), got dep: tls_self_signed_cert.k8s-test5-root
2016/11/22 00:15:00 [DEBUG] vertex provider.tls (close), got dep: tls_locally_signed_cert.k8s-test5-node
2016/11/22 00:15:00 [DEBUG] vertex provider.tls (close), got dep: tls_private_key.k8s-test5-node
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_master, got dep: tls_locally_signed_cert.k8s-test5-master
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_master, got dep: tls_locally_signed_cert.k8s-test5-node
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_master, got dep: vsphere_virtual_machine.kubevm2
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_master, got dep: provider.template
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_master, got dep: tls_self_signed_cert.k8s-test5-root
2016/11/22 00:15:00 [DEBUG] vertex null_resource.master, got dep: data.template_file.configure_master
2016/11/22 00:15:00 [DEBUG] vertex null_resource.master, got dep: provider.null
2016/11/22 00:15:00 [DEBUG] vertex null_resource.master, got dep: provisioner.remote-exec
2016/11/22 00:15:00 [DEBUG] vertex null_resource.master, got dep: provisioner.local-exec
2016/11/22 00:15:00 [DEBUG] vertex null_resource.master, got dep: data.template_file.cloudprovider
2016/11/22 00:15:00 [DEBUG] vertex null_resource.master, got dep: tls_locally_signed_cert.k8s-test5-admin
2016/11/22 00:15:00 [DEBUG] vertex null_resource.master, got dep: tls_private_key.k8s-test5-admin
2016/11/22 00:15:00 [DEBUG] vertex null_resource.master, got dep: tls_self_signed_cert.k8s-test5-root
2016/11/22 00:15:00 [DEBUG] vertex provisioner.remote-exec (close), got dep: null_resource.master
2016/11/22 00:15:00 [DEBUG] vertex data.template_file.configure_node, got dep: tls_locally_signed_cert.k8s-test5-master
2016/11/22 00:15:00 [DEBUG] vertex provider.template (close), got dep: data.template_file.configure_node
2016/11/22 00:15:00 [DEBUG] vertex provider.template (close), got dep: provider.template
2016/11/22 00:15:00 [DEBUG] vertex provider.template (close), got dep: data.template_file.cloudprovider
2016/11/22 00:15:00 [DEBUG] vertex provider.template (close), got dep: data.template_file.configure_master
2016/11/22 00:15:00 [DEBUG] vertex provider.null (close), got dep: null_resource.master
2016/11/22 00:15:00 [DEBUG] vertex provisioner.local-exec (close), got dep: null_resource.master
2016/11/22 00:15:00 [DEBUG] vertex null_resource.node2, got dep: data.template_file.configure_node
2016/11/22 00:15:00 [DEBUG] vertex null_resource.node2, got dep: data.template_file.cloudprovider
2016/11/22 00:15:00 [DEBUG] vertex null_resource.node2, got dep: provider.null
2016/11/22 00:15:00 [DEBUG] vertex provisioner.remote-exec (close), got dep: null_resource.node2
2016/11/22 00:15:00 [DEBUG] vertex provider.null (close), got dep: null_resource.node2
2016/11/22 00:15:00 [DEBUG] vertex root, got dep: provider.null (close)
2016/11/22 00:15:00 [DEBUG] vertex root, got dep: provider.template (close)
2016/11/22 00:15:00 [DEBUG] vertex root, got dep: provider.vsphere (close)
2016/11/22 00:15:00 [DEBUG] vertex root, got dep: provider.tls (close)
2016/11/22 00:15:00 [DEBUG] vertex root, got dep: provisioner.local-exec (close)
2016/11/22 00:15:00 [DEBUG] vertex root, got dep: provisioner.remote-exec (close)
2016/11/22 00:15:00 [DEBUG] plugin: waiting for all plugin processes to complete...
2016/11/22 00:15:00 [DEBUG] plugin: /bin/terraform: plugin process exited
2016/11/22 00:15:00 [DEBUG] plugin: /bin/terraform: plugin process exited
2016/11/22 00:15:00 [DEBUG] plugin: /bin/terraform: plugin process exited
2016/11/22 00:15:00 [DEBUG] plugin: /bin/terraform: plugin process exited
2016/11/22 00:15:00 [DEBUG] plugin: /bin/terraform: plugin process exited
2016/11/22 00:15:00 [DEBUG] plugin: /bin/terraform: plugin process exited



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
Makefile:53: recipe for target 'do' failed
make[1]: *** [do] Error 1
make[1]: Leaving directory '/opt/kubernetes-anywhere'
Makefile:41: recipe for target 'destroy-cluster' failed
make: *** [destroy-cluster] Error 2

[abrarshivani]

This is due to terraform destroy is flaky. Here is the same issue on terraform project: hashicorp/terraform#10365