Skip to content

Commit e1a8855

Browse files
RamLaviRamLavi
committed
blog: Load-balancer for vms on bare-metal K8s clusters
Signed-off-by: Ram Lavi <[email protected]>
1 parent 03ccf75 commit e1a8855

File tree

1 file changed

+241
-0
lines changed

1 file changed

+241
-0
lines changed

‎_posts/2022-04-03-Virtual-Machines-with-MetalLB.markdown

+241
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
---
2+
layout: post
3+
author: Ram Lavi
4+
title: Load-balancer for virtual machines on bare metal Kubernetes clusters
5+
description: This post illustrates setting up a virtual machine with MetalLb loadBalance service.
6+
navbar_active: Blogs
7+
pub-date: April 03
8+
pub-year: 2022
9+
category: news
10+
tags:
11+
[
12+
"Kubevirt",
13+
"kubernetes",
14+
"virtual machine",
15+
"VM",
16+
"load-balancer",
17+
"MetalLB",
18+
]
19+
comments: true
20+
---
21+
22+
## Introduction
23+
24+
Over the last year, Kubevirt and MetalLB have shown to be powerful duo in order to support fault-tolerant access to an application on virtual machines through an external IP address.
25+
As a Cluster administrator using an on-prem cluster without a network load-balancer, now it's possible to use MetalLB operator to provide load-balancer capabilities (with Services of type `LoadBalancer`) to virtual machines.
26+
27+
## Introducing MetalLB
28+
29+
[MetalLB](https://metallb.universe.tf/) allows you to create Kubernetes services of type `LoadBalancer`, and provides network load-balancer implementation in on-prem clusters that don’t run on a cloud provider.
30+
MetalLB is responsible for assigning/unassigning an external IP Address to your service, using IPs from pre-configured pools. In order for the external IPs to be announced externally, MetalLB works in 2 modes: Layer 2 and BGP:
31+
32+
- Layer 2 mode (ARP/NDP):
33+
34+
This mode - which actually does not implement real Load-balancing behavior - provides a failover mechanism where a single node owns the `LoadBalancer` service, until it fails, triggering another node to be chosen as the service owner. This configuration mode makes the IPs reachable from the local network.
35+
In this method, the MetalLB speaker pod announces the IPs in ARP (for IPv4) and NDP (for IPv6) protocols over the host network. From a network perspective, the node owning the service appears to have multiple IP addresses assigned to a network interface. After traffic is routed to the node, the service proxy sends the traffic to the application pods.
36+
37+
- BGP mode:
38+
39+
This mode provides real load-balancing behavior, by establishing BGP peering sessions with the network routers - which advertise the external IPs of the `LoadBalancer` service, distributing the load over the nodes.
40+
41+
To read more on MetalLB concepts, implementation and limitations, please read [its documentation](https://metallb.universe.tf/concepts/).
42+
43+
## Demo: Virtual machine with external IP and MetalLB load-balancer
44+
45+
With the following recipe we will end up with a nginx server running on a virtual machine, accessible outside the cluster using MetalLB load-balancer with Layer 2 mode.
46+
47+
### Demo environment setup
48+
49+
We are going to use [kind](https://kind.sigs.k8s.io) provider as an ephemeral Kubernetes cluster.
50+
To start it up follow this [installation guide](https://kind.sigs.k8s.io/docs/user/quick-start/#installation).
51+
52+
#### Prerequirement
53+
54+
This demo runs Docker on Linux, which allows sending traffic directly to the load-balancer's external IP if the IP space is within the docker IP space.
55+
On macOS and Windows however, docker does not expose the docker network to the host, rendering the external IP unreachable from other kind nodes. In order to workaround this, one could expose pods and services using extra port mappings as shown in the extra port mappings section of kind's [Configuration Guide](https://kind.sigs.k8s.io/docs/user/configuration#extra-port-mappings).
56+
57+
### Installing components
58+
59+
#### Installing MetalLB on the cluster
60+
61+
There are many ways to install MetalLB. For the sake of this example, we will install MetalLB operator on the cluster. To do this, please follow this [link](https://operatorhub.io/operator/metallb-operator).
62+
You can confirm the operator is installed by entering the following command (it may take a few seconds for the csv to appear):
63+
```bash
64+
kubectl get csv -n my-metallb-operator \
65+
-o custom-columns=Name:.metadata.name,Phase:.status.phase
66+
```
67+
Example output
68+
```bash
69+
Name Phase
70+
metallb-operator.v0.12.0 Succeeded
71+
```
72+
73+
After the operator in installed, we will create a MetalLB CR:
74+
```yaml
75+
cat <<EOF | kubectl apply -f -
76+
apiVersion: metallb.io/v1beta1
77+
kind: MetalLB
78+
metadata:
79+
name: metallb
80+
namespace: my-metallb-operator
81+
EOF
82+
```
83+
84+
#### Setting Address Pool to be used by the LoadBalancer
85+
86+
In order to complete the Layer 2 mode configuration, we need to set a range of IP Addresses for the LoadBalancer to use. We want this range to be on the docker kind network, so by using this command:
87+
88+
```bash
89+
docker network inspect -f '{{.IPAM.Config}}' kind
90+
```
91+
92+
You should get the subclass you can set the ip range from. The output should contain a cidr such as 172.18.1.0/16
93+
Using this result we will create the following Layer 2 address pool with 172.18.1.1-172.18.1.16 range:
94+
95+
```yaml
96+
cat <<EOF | kubectl apply -f -
97+
apiVersion: metallb.io/v1beta1
98+
kind: AddressPool
99+
metadata:
100+
name: addresspool-sample1
101+
namespace: my-metallb-operator
102+
spec:
103+
protocol: layer2
104+
addresses:
105+
- 172.18.1.1-172.18.1.16
106+
EOF
107+
```
108+
109+
#### Installing Kubevirt on the cluster
110+
111+
Following Kubevirt [user guide](https://kubevirt.io/user-guide/operations/installation/#installing-kubevirt-on-kubernetes) to install released version v0.51.0
112+
```bash
113+
export RELEASE=v0.51.0
114+
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml"
115+
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml"
116+
kubectl -n kubevirt wait kv kubevirt --timeout=360s --for condition=Available
117+
```
118+
119+
Now we have a Kubernetes cluster with all the pieces to start the Demo.
120+
121+
### Spin up a Virtual Machine running Nginx
122+
123+
Now it's time to start-up a virtual machine running nginx using this yaml:
124+
The virtual machine has a `metallb-service=nginx` we created to use when creating the service.
125+
```yaml
126+
cat <<EOF | kubectl apply -f -
127+
apiVersion: kubevirt.io/v1
128+
kind: VirtualMachine
129+
metadata:
130+
name: fedora-nginx
131+
namespace: default
132+
labels:
133+
metallb-service: nginx
134+
spec:
135+
running: true
136+
template:
137+
metadata:
138+
labels:
139+
metallb-service: nginx
140+
spec:
141+
domain:
142+
devices:
143+
disks:
144+
- disk:
145+
bus: virtio
146+
name: containerdisk
147+
- disk:
148+
bus: virtio
149+
name: cloudinitdisk
150+
interfaces:
151+
- masquerade: {}
152+
name: default
153+
resources:
154+
requests:
155+
memory: 1024M
156+
networks:
157+
- name: default
158+
pod: {}
159+
terminationGracePeriodSeconds: 0
160+
volumes:
161+
- containerDisk:
162+
image: kubevirt/fedora-cloud-container-disk-demo
163+
name: containerdisk
164+
- cloudInitNoCloud:
165+
userData: |-
166+
#cloud-config
167+
password: fedora
168+
chpasswd: { expire: False }
169+
packages:
170+
- nginx
171+
runcmd:
172+
- [ "systemctl", "enable", "--now", "nginx" ]
173+
name: cloudinitdisk
174+
EOF
175+
```
176+
177+
### Expose the virtual machine with a typed `LoaBalancer` service
178+
179+
When creating the `LoadBalancer` typed service, we need to remember annotating the address-pool we want to use
180+
`addresspool-sample1` and also add the selector `metallb-service: nginx`
181+
182+
```yaml
183+
cat <<EOF | kubectl apply -f -
184+
kind: Service
185+
apiVersion: v1
186+
metadata:
187+
name: metallb-nginx-svc
188+
namespace: default
189+
annotations:
190+
metallb.universe.tf/address-pool: addresspool-sample1
191+
spec:
192+
externalTrafficPolicy: Local
193+
ipFamilies:
194+
- IPv4
195+
ports:
196+
- name: tcp-5678
197+
protocol: TCP
198+
port: 5678
199+
targetPort: 80
200+
type: LoadBalancer
201+
selector:
202+
metallb-service: nginx
203+
EOF
204+
```
205+
206+
Notice that the service got assigned with an external IP from the range assigned by the PoolAddress:
207+
208+
```bash
209+
kubectl get service -n default metallb-nginx-svc
210+
```
211+
212+
Example output
213+
```bash
214+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
215+
metallb-nginx-svc LoadBalancer 10.96.254.136 172.18.1.1 5678:32438/TCP 53s
216+
```
217+
218+
### Access the virtual machine from outside the cluster
219+
220+
Finally, we can check that the nginx server is accessible from outside the cluster:
221+
```bash
222+
curl -s -o /dev/null 172.18.1.1:5678 && echo "URL exists"
223+
```
224+
225+
Example output
226+
```bash
227+
URL exists
228+
```
229+
230+
## Doing this on your own cluster
231+
232+
Moving outside the demo example, one who would like use MetalLB on their real life cluster, should also take other considerations in mind:
233+
- User privileges: you should have `cluster-admin` privileges on the cluster - in order to install MetalLB.
234+
- IP Ranges for MetalLB: getting IP Address pools allocation for MetalLB depends on your cluster environment:
235+
- If you're running a bare-metal cluster in a shared host environment, you need to first reserve this IP Address pool from your hosting provider.
236+
- Alternatively, if you're running on a private cluster, you can use one of the private IP Address spaces (a.k.a RFC1918 addresses). Such addresses are free, and work fine as long as you’re only providing cluster services to your LAN.
237+
238+
## Conclusion
239+
240+
In this blog post we used MetalLB to expose a service using an external IP assigned to a virtual machine.
241+
This illustrates how virtual machine traffic can be load-balanced via a service.

0 commit comments

Comments
 (0)