Skip to content

Don't create outbound LB if nat-gateway is enabled #1587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jackfrancis opened this issue Aug 6, 2021 · 5 comments · Fixed by #1589
Closed

Don't create outbound LB if nat-gateway is enabled #1587

jackfrancis opened this issue Aug 6, 2021 · 5 comments · Fixed by #1589
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@jackfrancis
Copy link
Contributor

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

Building a cluster w/ only natgateway "node" subnets included an outbound LB w/ the nodes in the backend pool

What did you expect to happen:

The natgateway resource itself handles outbound connections from nodes, so the outbound LB is superfluous (and in fact constrains the node scale, as we run out of outbound SNAT ports quickly compared to natgateway).

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • cluster-api-provider-azure version: v1alpha4
  • Kubernetes version: (use kubectl version): 1.21.3 (should repro for all versions)
  • OS (e.g. from /etc/os-release):
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 6, 2021
@dmeytin
Copy link

dmeytin commented Aug 8, 2021

@jackfrancis Jack, can you please explain how we'll able to define multiple node-pools that each pool will have a dedicated output IP? This is required when the cluster needs to handle more than 64K outbound ports and, thus, more than a single NAT Gateway is needed.

@jackfrancis
Copy link
Contributor Author

Because a NAT Gateway is a child property of a subnet, we'd need to support creating per-pool subnets, each with its own NAT Gateway.

@CecileRobertMichon @devigned is it possible to create multiple subnets, and assign specific node pools (i.e., MachineDeployment or MachinePool resources) to a specific subnet?

@CecileRobertMichon
Copy link
Contributor

Support was added in #1411, it's available in the latest CAPZ release

@dmeytin
Copy link

dmeytin commented Aug 10, 2021

So, no LB configuration is needed?

@jackfrancis
Copy link
Contributor Author

jackfrancis commented Aug 23, 2021

To clarify, the buggy behavior is that when you build a cluster with this template, you should only get a NAT Gateway to handle outbound SNAT:

https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/v0.5.2/templates/cluster-template-nat-gateway.yaml

Specifically, this declaration says "use NAT gateway to handle outbound SNAT for this node pool":

https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/v0.5.2/templates/cluster-template-nat-gateway.yaml#L42

What we see instead is that a Load Balancer specifically for handling outbound outbound SNAT is created in addition to the NAT Gateway.

To summarize, what's happening is that the decision whether or not to create the Outbound Load Balancer is based on the presence of a non-empty string 'Name' value of the subnet's NatGateway resource:

https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/v0.5.2/api/v1alpha4/types.go#L534

And we discovered that the way we're introspecting the NAT Gateway resource from the Azure API in fact produces an empty string Name value in the cluster-api spec, thus guaranteeing that the Outbound Load Balancer will always be created.

cc @CecileRobertMichon @shysank

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants