Skip to content

Commit 4ce0e1f

Browse files
RamLaviRamLavi
committed
blog: Load-balancer for vms on bare-metal K8s clusters
Signed-off-by: Ram Lavi <[email protected]>
1 parent 03ccf75 commit 4ce0e1f

File tree

1 file changed

+258
-0
lines changed

1 file changed

+258
-0
lines changed

‎_posts/2022-04-03-Virtual-Machines-with-MetalLB.markdown

+258
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,258 @@
1+
---
2+
layout: post
3+
author: Ram Lavi
4+
title: Load-balancer for virtual machines on bare metal Kubernetes clusters
5+
description: This post illustrates setting up a virtual machine with MetalLb loadBalance service.
6+
navbar_active: Blogs
7+
pub-date: April 03
8+
pub-year: 2022
9+
category: news
10+
tags:
11+
[
12+
"Kubevirt",
13+
"kubernetes",
14+
"virtual machine",
15+
"VM",
16+
"load-balancer",
17+
"MetalLB",
18+
]
19+
comments: true
20+
---
21+
22+
## Introduction
23+
24+
Over the last year, Kubevirt and MetalLB have shown to be powerful duo in order to support fault-tolerant access to an application on virtual machines through an external IP address.
25+
As a Cluster administrator using an on-prem cluster without a network load-balancer, now it's possible to use MetalLB operator to provide load-balancer capabilities (with Services of type `LoadBalancer`) to virtual machines.
26+
27+
## Introducing MetalLB
28+
29+
[MetalLB](https://metallb.universe.tf/) allows you to create Kubernetes services of type `LoadBalancer`, and provides network load-balancer implementation in on-prem clusters that don’t run on a cloud provider.
30+
MetalLB is responsible for assigning/unassigning an external IP Address to your service, using IPs from pre-configured pools. In order for the external IPs to be announced externally, MetalLB works in 2 modes: Layer 2 and BGP:
31+
32+
- Layer 2 mode (ARP/NDP):
33+
34+
This mode - which actually does not implement real Load-balancing behavior - provides a failover mechanism where a single node owns the `LoadBalancer` service, until it fails, triggering another node to be chosen as the service owner. This configuration mode makes the IPs reachable from the local network.
35+
In this method, the MetalLB speaker pod announces the IPs in ARP (for IPv4) and NDP (for IPv6) protocols over the host network. From a network perspective, the node owning the service appears to have multiple IP addresses assigned to a network interface. After traffic is routed to the node, the service proxy sends the traffic to the application pods.
36+
37+
- BGP mode:
38+
39+
This mode provides real load-balancing behavior, by establishing BGP peering sessions with the network routers - which advertise the external IPs of the `LoadBalancer` service, distributing the load over the nodes.
40+
41+
To read more on MetalLB concepts, implementation and limitations, please read [its documentation](https://metallb.universe.tf/concepts/).
42+
43+
## Demo: Virtual machine with external IP and MetalLB load-balancer
44+
45+
With the following recipe we will end up with a nginx server running on a virtual machine, accessible outside the cluster using MetalLB load-balancer with Layer 2 mode.
46+
47+
### Demo environment setup
48+
49+
We are going to use [kind](https://kind.sigs.k8s.io) provider as an ephemeral Kubernetes cluster.
50+
51+
#### Prerequirements
52+
53+
- First install kind on your machine following its [installation guide](https://kind.sigs.k8s.io/docs/user/quick-start/#installation).
54+
- To use kind, you will also need to [install docker](https://docs.docker.com/install/).
55+
56+
##### External IPs on macOS and Windows
57+
58+
This demo runs Docker on Linux, which allows sending traffic directly to the load-balancer's external IP if the IP space is within the docker IP space.
59+
On macOS and Windows however, docker does not expose the docker network to the host, rendering the external IP unreachable from other kind nodes. In order to workaround this, one could expose pods and services using extra port mappings as shown in the extra port mappings section of kind's [Configuration Guide](https://kind.sigs.k8s.io/docs/user/configuration#extra-port-mappings).
60+
61+
### Deploying cluster
62+
63+
To start a kind cluster:
64+
```bash
65+
kind create cluster
66+
```
67+
68+
Then to start using the cluster:
69+
```bash
70+
kubectl cluster-info --context kind-kind
71+
```
72+
73+
### Installing components
74+
75+
#### Installing MetalLB on the cluster
76+
77+
There are many ways to install MetalLB. For the sake of this example, we will install MetalLB operator on the cluster. To do this, please follow this [link](https://operatorhub.io/operator/metallb-operator).
78+
You can confirm the operator is installed by entering the following command (it may take a few seconds for the csv to appear):
79+
```bash
80+
kubectl get csv -n my-metallb-operator \
81+
-o custom-columns=Name:.metadata.name,Phase:.status.phase
82+
```
83+
Example output
84+
```bash
85+
Name Phase
86+
metallb-operator.v0.12.0 Succeeded
87+
```
88+
89+
After the operator in installed, we will create a MetalLB CR:
90+
```yaml
91+
cat <<EOF | kubectl apply -f -
92+
apiVersion: metallb.io/v1beta1
93+
kind: MetalLB
94+
metadata:
95+
name: metallb
96+
namespace: my-metallb-operator
97+
EOF
98+
```
99+
100+
#### Setting Address Pool to be used by the LoadBalancer
101+
102+
In order to complete the Layer 2 mode configuration, we need to set a range of IP addresses for the LoadBalancer to use.
103+
On Linux we can use the docker kind network (macOS and Windows users see [External IPs Prerequirement](#external-ips-on-macos-and-windows)), so by using this command:
104+
105+
```bash
106+
docker network inspect -f '{{.IPAM.Config}}' kind
107+
```
108+
109+
You should get the subclass you can set the IP range from. The output should contain a cidr such as 172.18.0.0/16
110+
Using this result we will create the following Layer 2 address pool with 172.18.1.1-172.18.1.16 range:
111+
112+
```yaml
113+
cat <<EOF | kubectl apply -f -
114+
apiVersion: metallb.io/v1beta1
115+
kind: AddressPool
116+
metadata:
117+
name: addresspool-sample1
118+
namespace: my-metallb-operator
119+
spec:
120+
protocol: layer2
121+
addresses:
122+
- 172.18.1.1-172.18.1.16
123+
EOF
124+
```
125+
126+
#### Installing Kubevirt on the cluster
127+
128+
Following Kubevirt [user guide](https://kubevirt.io/user-guide/operations/installation/#installing-kubevirt-on-kubernetes) to install released version v0.51.0
129+
```bash
130+
export RELEASE=v0.51.0
131+
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml"
132+
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml"
133+
kubectl -n kubevirt wait kv kubevirt --timeout=360s --for condition=Available
134+
```
135+
136+
Now we have a Kubernetes cluster with all the pieces to start the Demo.
137+
138+
### Spin up a Virtual Machine running Nginx
139+
140+
Now it's time to start-up a virtual machine running nginx using this yaml:
141+
The virtual machine has a `metallb-service=nginx` we created to use when creating the service.
142+
```yaml
143+
cat <<EOF | kubectl apply -f -
144+
apiVersion: kubevirt.io/v1
145+
kind: VirtualMachine
146+
metadata:
147+
name: fedora-nginx
148+
namespace: default
149+
labels:
150+
metallb-service: nginx
151+
spec:
152+
running: true
153+
template:
154+
metadata:
155+
labels:
156+
metallb-service: nginx
157+
spec:
158+
domain:
159+
devices:
160+
disks:
161+
- disk:
162+
bus: virtio
163+
name: containerdisk
164+
- disk:
165+
bus: virtio
166+
name: cloudinitdisk
167+
interfaces:
168+
- masquerade: {}
169+
name: default
170+
resources:
171+
requests:
172+
memory: 1024M
173+
networks:
174+
- name: default
175+
pod: {}
176+
terminationGracePeriodSeconds: 0
177+
volumes:
178+
- containerDisk:
179+
image: kubevirt/fedora-cloud-container-disk-demo
180+
name: containerdisk
181+
- cloudInitNoCloud:
182+
userData: |-
183+
#cloud-config
184+
password: fedora
185+
chpasswd: { expire: False }
186+
packages:
187+
- nginx
188+
runcmd:
189+
- [ "systemctl", "enable", "--now", "nginx" ]
190+
name: cloudinitdisk
191+
EOF
192+
```
193+
194+
### Expose the virtual machine with a typed `LoaBalancer` service
195+
196+
When creating the `LoadBalancer` typed service, we need to remember annotating the address-pool we want to use
197+
`addresspool-sample1` and also add the selector `metallb-service: nginx`
198+
199+
```yaml
200+
cat <<EOF | kubectl apply -f -
201+
kind: Service
202+
apiVersion: v1
203+
metadata:
204+
name: metallb-nginx-svc
205+
namespace: default
206+
annotations:
207+
metallb.universe.tf/address-pool: addresspool-sample1
208+
spec:
209+
externalTrafficPolicy: Local
210+
ipFamilies:
211+
- IPv4
212+
ports:
213+
- name: tcp-5678
214+
protocol: TCP
215+
port: 5678
216+
targetPort: 80
217+
type: LoadBalancer
218+
selector:
219+
metallb-service: nginx
220+
EOF
221+
```
222+
223+
Notice that the service got assigned with an external IP from the range assigned by the PoolAddress:
224+
225+
```bash
226+
kubectl get service -n default metallb-nginx-svc
227+
```
228+
229+
Example output
230+
```bash
231+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
232+
metallb-nginx-svc LoadBalancer 10.96.254.136 172.18.1.1 5678:32438/TCP 53s
233+
```
234+
235+
### Access the virtual machine from outside the cluster
236+
237+
Finally, we can check that the nginx server is accessible from outside the cluster:
238+
```bash
239+
curl -s -o /dev/null 172.18.1.1:5678 && echo "URL exists"
240+
```
241+
242+
Example output
243+
```bash
244+
URL exists
245+
```
246+
247+
## Doing this on your own cluster
248+
249+
Moving outside the demo example, one who would like use MetalLB on their real life cluster, should also take other considerations in mind:
250+
- User privileges: you should have `cluster-admin` privileges on the cluster - in order to install MetalLB.
251+
- IP Ranges for MetalLB: getting IP Address pools allocation for MetalLB depends on your cluster environment:
252+
- If you're running a bare-metal cluster in a shared host environment, you need to first reserve this IP Address pool from your hosting provider.
253+
- Alternatively, if you're running on a private cluster, you can use one of the private IP Address spaces (a.k.a RFC1918 addresses). Such addresses are free, and work fine as long as you’re only providing cluster services to your LAN.
254+
255+
## Conclusion
256+
257+
In this blog post we used MetalLB to expose a service using an external IP assigned to a virtual machine.
258+
This illustrates how virtual machine traffic can be load-balanced via a service.

0 commit comments

Comments
 (0)