Skip to content

Commit 50df1f2

Browse files
RamLaviRamLavi
committed
blog: Load-balancer for vms on bare-metal K8s clusters
Signed-off-by: Ram Lavi <[email protected]>
1 parent 03ccf75 commit 50df1f2

File tree

1 file changed

+246
-0
lines changed

1 file changed

+246
-0
lines changed

‎_posts/2022-04-03-Virtual-Machines-with-MetalLB.markdown

+246
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
---
2+
layout: post
3+
author: Ram Lavi
4+
title: Load-balancer for virtual machines on bare metal Kubernetes clusters
5+
description: This post illustrates setting up a virtual machine with MetalLB loadBalance service.
6+
navbar_active: Blogs
7+
pub-date: May 03
8+
pub-year: 2022
9+
category: news
10+
tags:
11+
[
12+
"Kubevirt",
13+
"kubernetes",
14+
"virtual machine",
15+
"VM",
16+
"load-balancer",
17+
"MetalLB",
18+
]
19+
comments: true
20+
---
21+
22+
## Introduction
23+
24+
Over the last year, Kubevirt and MetalLB have shown to be powerful duo in order to support fault-tolerant access to an application on virtual machines through an external IP address.
25+
As a Cluster administrator using an on-prem cluster without a network load-balancer, now it's possible to use MetalLB operator to provide load-balancer capabilities (with Services of type `LoadBalancer`) to virtual machines.
26+
27+
## Introducing MetalLB
28+
29+
[MetalLB](https://metallb.universe.tf/) allows you to create Kubernetes services of type `LoadBalancer`, and provides network load-balancer implementation in on-prem clusters that don’t run on a cloud provider.
30+
MetalLB is responsible for assigning/unassigning an external IP Address to your service, using IPs from pre-configured pools. In order for the external IPs to be announced externally, MetalLB works in 2 modes: Layer 2 and BGP:
31+
32+
- Layer 2 mode (ARP/NDP):
33+
34+
This mode - which actually does not implement real Load-balancing behavior - provides a failover mechanism where a single node owns the `LoadBalancer` service, until it fails, triggering another node to be chosen as the service owner. This configuration mode makes the IPs reachable from the local network.
35+
In this method, the MetalLB speaker pod announces the IPs in ARP (for IPv4) and NDP (for IPv6) protocols over the host network. From a network perspective, the node owning the service appears to have multiple IP addresses assigned to a network interface. After traffic is routed to the node, the service proxy sends the traffic to the application pods.
36+
37+
- BGP mode:
38+
39+
This mode provides real load-balancing behavior, by establishing BGP peering sessions with the network routers - which advertise the external IPs of the `LoadBalancer` service, distributing the load over the nodes.
40+
41+
To read more on MetalLB concepts, implementation and limitations, please read [its documentation](https://metallb.universe.tf/concepts/).
42+
43+
## Demo: Virtual machine with external IP and MetalLB load-balancer
44+
45+
With the following recipe we will end up with a nginx server running on a virtual machine, accessible outside the cluster using MetalLB load-balancer with Layer 2 mode.
46+
47+
### Demo environment setup
48+
49+
We are going to use [kind](https://kind.sigs.k8s.io) provider as an ephemeral Kubernetes cluster.
50+
51+
#### Prerequirements
52+
53+
- First install kind on your machine following its [installation guide](https://kind.sigs.k8s.io/docs/user/quick-start/#installation).
54+
- To use kind, you will also need to [install docker](https://docs.docker.com/install/).
55+
56+
##### External IPs on macOS and Windows
57+
58+
This demo runs Docker on Linux, which allows sending traffic directly to the load-balancer's external IP if the IP space is within the docker IP space.
59+
On macOS and Windows however, docker does not expose the docker network to the host, rendering the external IP unreachable from other kind nodes. In order to workaround this, one could expose pods and services using extra port mappings as shown in the extra port mappings section of kind's [Configuration Guide](https://kind.sigs.k8s.io/docs/user/configuration#extra-port-mappings).
60+
61+
### Deploying cluster
62+
63+
To start a kind cluster:
64+
```bash
65+
kind create cluster
66+
```
67+
68+
In order to interact with the specific cluster created:
69+
```bash
70+
kubectl cluster-info --context kind-kind
71+
```
72+
73+
### Installing components
74+
75+
#### Installing MetalLB on the cluster
76+
77+
There are [many ways](https://metallb.universe.tf/installation/) to install MetalLB. For the sake of this example, we will install MetalLB via Manifests. To do this, follow this [guide](https://metallb.universe.tf/installation/#installation-by-manifest).
78+
Confirm successful installation by waiting for MetalLB pods to have a status of Running:
79+
```bash
80+
kubectl get pods -n metallb-system --watch
81+
```
82+
83+
#### Setting Address Pool to be used by the LoadBalancer
84+
85+
In order to complete the Layer 2 mode configuration, we need to set a range of IP addresses for the LoadBalancer to use.
86+
On Linux we can use the docker kind network (macOS and Windows users see [External IPs Prerequirement](#external-ips-on-macos-and-windows)), so by using this command:
87+
88+
```bash
89+
docker network inspect -f '{{.IPAM.Config}}' kind
90+
```
91+
92+
You should get the subclass you can set the IP range from. The output should contain a cidr such as 172.18.0.0/16
93+
Using this result we will create the following Layer 2 address pool with 172.18.1.1-172.18.1.16 range:
94+
95+
```yaml
96+
cat <<EOF | kubectl apply -f -
97+
apiVersion: v1
98+
kind: ConfigMap
99+
metadata:
100+
namespace: metallb-system
101+
name: config
102+
data:
103+
config: |
104+
address-pools:
105+
- name: addresspool-sample1
106+
protocol: layer2
107+
addresses:
108+
- 172.18.1.1-172.18.1.16
109+
EOF
110+
```
111+
112+
#### Installing Kubevirt on the cluster
113+
114+
Following Kubevirt [user guide](https://kubevirt.io/user-guide/operations/installation/#installing-kubevirt-on-kubernetes) to install released version v0.51.0
115+
```bash
116+
export RELEASE=v0.51.0
117+
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml"
118+
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml"
119+
kubectl -n kubevirt wait kv kubevirt --timeout=360s --for condition=Available
120+
```
121+
122+
Now we have a Kubernetes cluster with all the pieces to start the Demo.
123+
124+
### Spin up a Virtual Machine running Nginx
125+
126+
Now it's time to start-up a virtual machine running nginx using this yaml:
127+
The virtual machine has a `metallb-service=nginx` we created to use when creating the service.
128+
```yaml
129+
cat <<EOF | kubectl apply -f -
130+
apiVersion: kubevirt.io/v1
131+
kind: VirtualMachine
132+
metadata:
133+
name: fedora-nginx
134+
namespace: default
135+
labels:
136+
metallb-service: nginx
137+
spec:
138+
running: true
139+
template:
140+
metadata:
141+
labels:
142+
metallb-service: nginx
143+
spec:
144+
domain:
145+
devices:
146+
disks:
147+
- disk:
148+
bus: virtio
149+
name: containerdisk
150+
- disk:
151+
bus: virtio
152+
name: cloudinitdisk
153+
interfaces:
154+
- masquerade: {}
155+
name: default
156+
resources:
157+
requests:
158+
memory: 1024M
159+
networks:
160+
- name: default
161+
pod: {}
162+
terminationGracePeriodSeconds: 0
163+
volumes:
164+
- containerDisk:
165+
image: kubevirt/fedora-cloud-container-disk-demo
166+
name: containerdisk
167+
- cloudInitNoCloud:
168+
userData: |-
169+
#cloud-config
170+
password: fedora
171+
chpasswd: { expire: False }
172+
packages:
173+
- nginx
174+
runcmd:
175+
- [ "systemctl", "enable", "--now", "nginx" ]
176+
name: cloudinitdisk
177+
EOF
178+
```
179+
180+
### Expose the virtual machine with a typed `LoaBalancer` service
181+
182+
When creating the `LoadBalancer` typed service, we need to remember annotating the address-pool we want to use
183+
`addresspool-sample1` and also add the selector `metallb-service: nginx`
184+
185+
```yaml
186+
cat <<EOF | kubectl apply -f -
187+
kind: Service
188+
apiVersion: v1
189+
metadata:
190+
name: metallb-nginx-svc
191+
namespace: default
192+
annotations:
193+
metallb.universe.tf/address-pool: addresspool-sample1
194+
spec:
195+
externalTrafficPolicy: Local
196+
ipFamilies:
197+
- IPv4
198+
ports:
199+
- name: tcp-5678
200+
protocol: TCP
201+
port: 5678
202+
targetPort: 80
203+
type: LoadBalancer
204+
selector:
205+
metallb-service: nginx
206+
EOF
207+
```
208+
209+
Notice that the service got assigned with an external IP from the range assigned by the PoolAddress:
210+
211+
```bash
212+
kubectl get service -n default metallb-nginx-svc
213+
```
214+
215+
Example output
216+
```bash
217+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
218+
metallb-nginx-svc LoadBalancer 10.96.254.136 172.18.1.1 5678:32438/TCP 53s
219+
```
220+
221+
### Access the virtual machine from outside the cluster
222+
223+
Finally, we can check that the nginx server is accessible from outside the cluster:
224+
```bash
225+
curl -s -o /dev/null 172.18.1.1:5678 && echo "URL exists"
226+
```
227+
228+
Example output
229+
```bash
230+
URL exists
231+
```
232+
Note that it may take a short while for the URL to work after setting the service.
233+
234+
235+
## Doing this on your own cluster
236+
237+
Moving outside the demo example, one who would like use MetalLB on their real life cluster, should also take other considerations in mind:
238+
- User privileges: you should have `cluster-admin` privileges on the cluster - in order to install MetalLB.
239+
- IP Ranges for MetalLB: getting IP Address pools allocation for MetalLB depends on your cluster environment:
240+
- If you're running a bare-metal cluster in a shared host environment, you need to first reserve this IP Address pool from your hosting provider.
241+
- Alternatively, if you're running on a private cluster, you can use one of the private IP Address spaces (a.k.a RFC1918 addresses). Such addresses are free, and work fine as long as you’re only providing cluster services to your LAN.
242+
243+
## Conclusion
244+
245+
In this blog post we used MetalLB to expose a service using an external IP assigned to a virtual machine.
246+
This illustrates how virtual machine traffic can be load-balanced via a service.

0 commit comments

Comments
 (0)