Error updating service port

[alapidas]

I created a LoadBalancer service and tried to change the port, and CCM errored out with the following:

I1031 21:46:33.332339       1 service_controller.go:286] Ensuring LB for service default/frontend
I1031 21:46:33.371770       1 load_balancer.go:252] Applying `delete` action on backend set `TCP-80` for lb `ocid1.loadbalancer.oc1.iad.aaaaaaaaxsq3zcrkxgu4rvkmc75iw27jme6zz6xijesw52qr26fnlz467wtq`
E1031 21:46:33.416808       1 service_controller.go:749] Failed to process service. Retrying in 5s: Failed to ensure load balancer for service default/frontend: update backendsets: update lb security list rules `ocid1.securitylist.oc1.iad.aaaaaaaa6tb2ljjdxvgxlpcs7wrfvi6oinv3c5cjn2oujcxyvuvu45dtl6eq` for subnet `ocid1.subnet.oc1.iad.aaaaaaaawt2lzrjd2dpoj3mvi5ljznqxaem7fzjjuffp5ac36mc2uw2td5vq: Status: 412; Code: NoEtagMatch; OPC Request ID: /27241DEF8077EF887A5AE78FF3F255C9/BA7DF04E4B9046F7BBC883B3BA02E36A; Message: Entity security list with ID ocid1.securitylist.oc1.iad.aaaaaaaa6tb2ljjdxvgxlpcs7wrfvi6oinv3c5cjn2oujcxyvuvu45dtl6eq has a computed tag of 66939e3f, but is passed a tag of 11f4c135
I1031 21:46:38.416995       1 service_controller.go:286] Ensuring LB for service default/frontend
I1031 21:46:38.438356       1 load_balancer.go:252] Applying `delete` action on backend set `TCP-80` for lb `ocid1.loadbalancer.oc1.iad.aaaaaaaaxsq3zcrkxgu4rvkmc75iw27jme6zz6xijesw52qr26fnlz467wtq`
E1031 21:46:51.708515       1 service_controller.go:749] Failed to process service. Retrying in 10s: Failed to ensure load balancer for service default/frontend: update backendsets: WorkRequest "ocid1.loadbalancerworkrequest.oc1.iad.aaaaaaaaqzsol2b24vbgcazcpggnwe2i5q2rt2rf4m4z6wmow6exjfha6xla" failed: Listener 'TCP-80' has unknown default backend set 'TCP-80'
I1031 21:47:01.708678       1 service_controller.go:286] Ensuring LB for service default/frontend
I1031 21:47:01.729085       1 load_balancer.go:252] Applying `delete` action on backend set `TCP-80` for lb `ocid1.loadbalancer.oc1.iad.aaaaaaaaxsq3zcrkxgu4rvkmc75iw27jme6zz6xijesw52qr26fnlz467wtq`
E1031 21:47:10.120603       1 service_controller.go:749] Failed to process service. Retrying in 20s: Failed to ensure load balancer for service default/frontend: update backendsets: WorkRequest "ocid1.loadbalancerworkrequest.oc1.iad.aaaaaaaawj6s3ohnsmbzy4ijc6ratb4etrriahe4myr2i5d5p342bznowqha" failed: Listener 'TCP-80' has unknown default backend set 'TCP-80'
I1031 21:47:30.120776       1 service_controller.go:286] Ensuring LB for service default/frontend
I1031 21:47:30.137829       1 load_balancer.go:252] Applying `delete` action on backend set `TCP-80` for lb `ocid1.loadbalancer.oc1.iad.aaaaaaaaxsq3zcrkxgu4rvkmc75iw27jme6zz6xijesw52qr26fnlz467wtq`
E1031 21:47:43.705711       1 service_controller.go:749] Failed to process service. Retrying in 40s: Failed to ensure load balancer for service default/frontend: update backendsets: WorkRequest "ocid1.loadbalancerworkrequest.oc1.iad.aaaaaaaao4n6tmc2zudmjkmtjnt3v4nyac4wusxjtammyvp453tyuisnenkq" failed: Listener 'TCP-80' has unknown default backend set 'TCP-80'

And in the OCI Console:

[prydie]

Reproduced with higher verbosity logging on (-v=4):

I1101 09:55:24.726619       1 service_controller.go:286] Ensuring LB for service default/echoheaders
I1101 09:55:24.726675       1 load_balancer.go:164] Ensure load balancer '18c72da4-beea-11e7-a411-00001701ce75' called for 'echoheaders' with 2 nodes.
I1101 09:55:24.757144       1 load_balancer_spec.go:156] No SSL enabled ports found for service "echoheaders"
I1101 09:55:24.757196       1 load_balancer.go:252] Applying `delete` action on backend set `TCP-80` for lb `ocid1.loadbalancer.oc1.phx.aaaaaaaatizvm3l46a6yaab74urov2riflh6tnxmxqvb5xza3vtoe7yfbdua`
E1101 09:55:24.847130       1 service_controller.go:749] Failed to process service. Retrying in 5s: Failed to ensure load balancer for service default/echoheaders: update backendsets: update lb security list rules `ocid1.securitylist.oc1.phx.aaaaaaaak44nwlvzh3yon7j3pyoxjd45pef7jfcmv24djubpj3n6p4x7pywa` for subnet `ocid1.subnet.oc1.phx.aaaaaaaasa53hlkzk6nzksqfccegk2qnkxmphkblst3riclzs4rhwg7rg57q: Status: 412; Code: NoEtagMatch; OPC Request ID: /DBFC53BEB8735D8F04318A6E3787100F/29FF1EA52D2D499F8D227A39E36B2417; Message: Entity security list with ID ocid1.securitylist.oc1.phx.aaaaaaaak44nwlvzh3yon7j3pyoxjd45pef7jfcmv24djubpj3n6p4x7pywa has a computed tag of 67f10de0, but is passed a tag of bda19f5b
I1101 09:55:24.847173       1 service_controller.go:723] Finished syncing service "default/echoheaders" (120.566772ms)
I1101 09:55:25.407255       1 leaderelection.go:199] successfully renewed lease kube-system/cloud-controller-manager
I1101 09:55:25.675962       1 reflector.go:286] github.com/oracle/oci-cloud-controller-manager/vendor/k8s.io/client-go/informers/factory.go:73: forcing resync
I1101 09:55:27.411455       1 leaderelection.go:199] successfully renewed lease kube-system/cloud-controller-manager
I1101 09:55:29.415174       1 leaderelection.go:199] successfully renewed lease kube-system/cloud-controller-manager
I1101 09:55:29.847376       1 service_controller.go:286] Ensuring LB for service default/echoheaders
I1101 09:55:29.847530       1 load_balancer.go:164] Ensure load balancer '18c72da4-beea-11e7-a411-00001701ce75' called for 'echoheaders' with 2 nodes.
I1101 09:55:29.874035       1 load_balancer_spec.go:156] No SSL enabled ports found for service "echoheaders"
I1101 09:55:29.874096       1 load_balancer.go:252] Applying `delete` action on backend set `TCP-80` for lb `ocid1.loadbalancer.oc1.phx.aaaaaaaatizvm3l46a6yaab74urov2riflh6tnxmxqvb5xza3vtoe7yfbdua`
I1101 09:55:29.946231       1 load_balancer_security_lists.go:137] No changes for lb subnet security list `ocid1.securitylist.oc1.phx.aaaaaaaak44nwlvzh3yon7j3pyoxjd45pef7jfcmv24djubpj3n6p4x7pywa`
I1101 09:55:30.286191       1 client.go:367] Polling WorkRequest "ocid1.loadbalancerworkrequest.oc1.phx.aaaaaaaaavacqupztfjf7mxzqyelmfdtylbnw7le3wj6err237ni6yf2m6sq"...
I1101 09:55:30.870108       1 client.go:377] WorkRequest "ocid1.loadbalancerworkrequest.oc1.phx.aaaaaaaaavacqupztfjf7mxzqyelmfdtylbnw7le3wj6err237ni6yf2m6sq" state: 'ACCEPTED'
I1101 09:55:31.418500       1 leaderelection.go:199] successfully renewed lease kube-system/cloud-controller-manager
I1101 09:55:33.422482       1 leaderelection.go:199] successfully renewed lease kube-system/cloud-controller-manager
I1101 09:55:33.707779       1 client.go:377] WorkRequest "ocid1.loadbalancerworkrequest.oc1.phx.aaaaaaaaavacqupztfjf7mxzqyelmfdtylbnw7le3wj6err237ni6yf2m6sq" state: 'ACCEPTED'
I1101 09:55:35.426142       1 leaderelection.go:199] successfully renewed lease kube-system/cloud-controller-manager
I1101 09:55:36.973114       1 client.go:377] WorkRequest "ocid1.loadbalancerworkrequest.oc1.phx.aaaaaaaaavacqupztfjf7mxzqyelmfdtylbnw7le3wj6err237ni6yf2m6sq" state: 'FAILED'
E1101 09:55:36.973192       1 service_controller.go:749] Failed to process service. Retrying in 10s: Failed to ensure load balancer for service default/echoheaders: update backendsets: WorkRequest "ocid1.loadbalancerworkrequest.oc1.phx.aaaaaaaaavacqupztfjf7mxzqyelmfdtylbnw7le3wj6err237ni6yf2m6sq" failed: Listener 'TCP-80' has unknown default backend set 'TCP-80'
I1101 09:55:36.973212       1 service_controller.go:723] Finished syncing service "default/echoheaders" (7.125849086s)

There are potentially two problems here:

• Firstly, the etag error being thrown in the first place, however, this may still be expected behavior (cc @jhorwit2).

• Secondly, we DELETE the BackendSet corresponding the old port prior to deleting its associated Listener which has it's default BackendSet set to the BackendSet we are deleting. This would lead to the Listener having an invalid reference to a BackendSet that no longer exists so OCI rejects the WorkRequest. We previously handled changes to the front-end port up-front which avoided this, however, with the refactor to simplify the handling of LB updates this must have slipped by.

  The core problem here is that Listeners and Backend sets must be created and deleted in the opposite order due to this relationship which the refactored LB update algorithm doesn't take into account. When deleting a service of type: LoadBalancer we issue a delete to the LoadBalancer itself which does a cascading delete of the components so this issue isn't hit.