Panic in CCM

[alapidas]

I deployed a LB service into my cluster before any workers existed, and then observed a panic in CCM (0.3.1)

I0406 15:37:25.273352       1 client.go:165] configuring OCI client with a new trusted ca: /etc/oke/certs/pp/cacert
I0406 15:37:25.274238       1 client.go:165] configuring OCI client with a new trusted ca: /etc/oke/certs/pp/cacert
I0406 15:37:25.275065       1 client.go:165] configuring OCI client with a new trusted ca: /etc/oke/certs/pp/cacert
I0406 15:37:25.275283       1 client.go:108] No vcn provided in cloud provider config. Falling back to looking up VCN via LB subnet.
W0406 15:37:25.376441       1 authentication.go:55] Authentication is disabled
I0406 15:37:25.376489       1 insecure_serving.go:44] Serving insecurely on [::]:10253
I0406 15:37:25.376948       1 leaderelection.go:174] attempting to acquire leader lease...
I0406 15:37:57.385837       1 leaderelection.go:184] successfully acquired lease kube-system/cloud-controller-manager
I0406 15:37:57.386229       1 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"a196cb3b-39af-11e8-97e4-0a580aed603c", APIVersion:"v1", ResourceVersion:"834", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' oke-tkm-al-c2dsnjuha2d-8b7b77d99-vw6h6_67d3ebc0-39b0-11e8-8f29-0a580aed603c became leader
I0406 15:37:57.386285       1 ccm.go:102] Waiting for node informer cache to sync
I0406 15:37:57.486529       1 ccm.go:114] Waiting for service informer cache to sync
I0406 15:37:57.587429       1 node_controller.go:86] Sending events to api server.
I0406 15:37:57.588092       1 pvlcontroller.go:107] Starting PersistentVolumeLabelController
I0406 15:37:57.588113       1 controller_utils.go:1019] Waiting for caches to sync for persistent volume label controller
I0406 15:37:57.588416       1 controllermanager.go:258] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
I0406 15:37:57.588534       1 service_controller.go:183] Starting service controller
I0406 15:37:57.588545       1 controller_utils.go:1019] Waiting for caches to sync for service controller
I0406 15:37:57.688309       1 controller_utils.go:1026] Caches are synced for persistent volume label controller
I0406 15:37:57.688795       1 controller_utils.go:1026] Caches are synced for service controller
I0406 15:37:57.935423       1 load_balancer.go:89] Load balancer "0a4f5c0f-39b0-11e8-97e4-0a580aed603c" does not exist
I0406 15:37:57.935474       1 service_controller.go:326] Not persisting unchanged LoadBalancerStatus for service kube-system/tiller-deploy to registry.
I0406 15:37:58.133884       1 load_balancer.go:89] Load balancer "0aa0c8aa-39b0-11e8-97e4-0a580aed603c" does not exist
I0406 15:37:58.133920       1 service_controller.go:326] Not persisting unchanged LoadBalancerStatus for service kube-system/kubernetes-dashboard to registry.
I0406 15:37:58.890456       1 load_balancer.go:89] Load balancer "560732ab-39b0-11e8-bae2-0a580aed3f9a" does not exist
I0406 15:37:58.890491       1 service_controller.go:326] Not persisting unchanged LoadBalancerStatus for service default/mysql to registry.
I0406 15:37:58.890535       1 service_controller.go:300] Ensuring LB for service default/wordpress
I0406 15:37:58.890991       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"wordpress", UID:"566f9443-39b0-11e8-bae2-0a580aed3f9a", APIVersion:"v1", ResourceVersion:"726", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0406 15:37:58.891045       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"wordpress", UID:"566f9443-39b0-11e8-bae2-0a580aed3f9a", APIVersion:"v1", ResourceVersion:"726", FieldPath:""}): type: 'Warning' reason: 'UnAvailableLoadBalancer' There are no available nodes for LoadBalancer service default/wordpress
W0406 15:37:59.055318       1 load_balancer_util.go:261] BackendSet "TCP-80" has no Backends
W0406 15:37:59.055354       1 load_balancer_util.go:246] BackendSet "TCP-80" has no Backends
I0406 15:37:59.207187       1 load_balancer.go:398] Applying "update" action on backend set "TCP-80" for lb "ocid1.loadbalancer.oc1.phx.aaaaaaaamxm4uhcw3fuvttyzgs3aeqpe7vfwfrbnyebzrl4nlqzs2qfzwogq" (ports={ListenerPort:0 BackendPort:0 HealthCheckerPort:10256})
E0406 15:37:59.385925       1 runtime.go:66] Observed a panic: &runtime.TypeAssertionError{interfaceString:"interface {}", concreteString:"", assertedString:"[]interface {}", missingMethod:""} (interface conversion: interface {} is nil, not []interface {})
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:509
/usr/local/go/src/runtime/panic.go:491
/usr/local/go/src/runtime/iface.go:172
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:177
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:168
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:212
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:235
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:465
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:500
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:527
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/loadbalancer/loadbalancer_client.go:588
/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/oci/client/load_balancer.go:187
/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/oci/load_balancer.go:412
/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/oci/load_balancer.go:363
/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/oci/load_balancer.go:328
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:374
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:305
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:252
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:732
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:213
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:202
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:191
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:2337
panic: interface conversion: interface {} is nil, not []interface {} [recovered]
	panic: interface conversion: interface {} is nil, not []interface {}

goroutine 168 [running]:
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111
panic(0x16bba80, 0xc4207506c0)
	/usr/local/go/src/runtime/panic.go:491 +0x283
github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common.omitNilFieldsInJSON(0x0, 0x0, 0x15bb640, 0xc4207c8900, 0x97, 0x15bb640, 0x1554da1, 0x20, 0x0)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:177 +0xe90
github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common.omitNilFieldsInJSON(0x1691480, 0xc4204bdb60, 0x17d1f00, 0xc4207c8900, 0x99, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:168 +0xaba
github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common.removeNilFieldsInJSONWithTaggedStruct(0xc42038b540, 0x116, 0x124, 0x17d1f00, 0xc4207c8900, 0x99, 0x0, 0x17d2000, 0x0, 0x155b0c1, ...)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:212 +0xf5
github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common.addToBody(0xc4207c3210, 0x17d1f00, 0xc4207c8900, 0x99, 0x155b0c1, 0x17, 0x0, 0x0, 0x24cef80, 0x17d1f00, ...)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:235 +0x1e2
github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common.structToRequestPart(0xc4207c3210, 0x17d2000, 0xc4207c8900, 0x99, 0x17d2000, 0x12)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:465 +0x3d5
github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common.HTTPRequestMarshaller(0x17d2000, 0xc4207c8900, 0xc4207c3210, 0x3c, 0x18d525d)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:500 +0x16b
github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common.MakeDefaultHTTPRequestWithTaggedStruct(0x18d525d, 0x3, 0x1927427, 0x3c, 0x17d2000, 0xc4207c8900, 0x18d525d, 0x3, 0xc420406900, 0x18d9cb0, ...)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/common/http.go:527 +0xf0
github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/loadbalancer.LoadBalancerClient.UpdateBackendSet(0x24a4920, 0xc4203692f0, 0x24a5d60, 0xc420255720, 0x0, 0xc4203cede0, 0x21, 0xc4203ceae0, 0x2c, 0x18d9ac0, ...)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/github.com/oracle/oci-go-sdk/loadbalancer/loadbalancer_client.go:588 +0x14e
github.com/oracle/oci-cloud-controller-manager/pkg/oci/client.(*client).UpdateBackendSet(0xc420245f40, 0x24c3ca0, 0xc420092030, 0xc4200809c0, 0x57, 0xc4202f44a0, 0x6, 0xc420696500, 0xc4202e5010, 0x0, ...)
	/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/oci/client/load_balancer.go:187 +0x1d2
github.com/oracle/oci-cloud-controller-manager/pkg/oci.(*CloudProvider).updateBackendSet(0xc420202820, 0x24c3ca0, 0xc420092030, 0xc4200809c0, 0x57, 0xc42039c120, 0xc4202e50b0, 0x2, 0x2, 0x0, ...)
	/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/oci/load_balancer.go:412 +0x58c
github.com/oracle/oci-cloud-controller-manager/pkg/oci.(*CloudProvider).updateLoadBalancer(0xc420202820, 0x24c3ca0, 0xc420092030, 0xc42039c090, 0xc420364aa0, 0x0, 0x2)
	/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/oci/load_balancer.go:363 +0x3dd
github.com/oracle/oci-cloud-controller-manager/pkg/oci.(*CloudProvider).EnsureLoadBalancer(0xc420202820, 0x24c3ca0, 0xc420092030, 0x18dc708, 0xa, 0xc4207905a0, 0x0, 0x0, 0x0, 0xc4200a28e0, ...)
	/go/src/github.com/oracle/oci-cloud-controller-manager/pkg/oci/load_balancer.go:328 +0x4f7
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).ensureLoadBalancer(0xc42067a000, 0xc4207905a0, 0xc4207905a0, 0x18d7322, 0x6)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:374 +0xe8
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).createLoadBalancerIfNeeded(0xc42067a000, 0xc4204c2240, 0x11, 0xc4207905a0, 0xc42021de28, 0xc4207c3cc0)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:305 +0x20e
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).processServiceUpdate(0xc42067a000, 0xc420690008, 0xc4207905a0, 0xc4204c2240, 0x11, 0x0, 0x0)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:252 +0xe9
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).syncService(0xc42067a000, 0xc4204c2240, 0x11, 0x0, 0x0)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:732 +0x30d
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).processNextWorkItem(0xc42067a000, 0x62f300)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:213 +0xe1
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).worker(0xc42067a000)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:202 +0x2b
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).(github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.worker)-fm()
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:191 +0x2a
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4203c0570)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4203c0570, 0x3b9aca00, 0x0, 0x1, 0xc4201b4420)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4203c0570, 0x3b9aca00, 0xc4201b4420)
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service.(*ServiceController).Run
	/go/src/github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/service/service_controller.go:191 +0x20c