Skip to content

Latest commit

 

History

History
264 lines (209 loc) · 13.3 KB

README.md

File metadata and controls

264 lines (209 loc) · 13.3 KB

HomeLab

HomeLab Server/Cluster, Virtual Sandbox Cluster, & Desktop Configuration

Table of Contents

Remote VsCode

 ansible-playbook -i ansible/inventory --ask-become-pass ansible/vscode-server.yaml --ask-pass \
  -e 'ansible_python_interpreter=/usr/bin/python3'

Server

graph TD
 linkStyle default interpolate basis

subgraph Networking
wan1[<center>WAN<br>192.168.100.1</center>]---|1000/50 Mb|router{<center>UDM Pro SE<br>10.0.0.1</center>}
wan2[<center>LTE<br>192.168.1.1</center>]---|750/25 Mb|router
router---|10GbE|switch1[<center>USW-Pro-Max-16<br></center>]
router---|2.5GbE|ap1{<center>U7 Pro Wall<br></center>}
switch1---|2.5GbE|ap2{<center>U7 Pro Max<br></center>}
switch1----|10GbE|switch10[<center>USW-Aggregation<br></center>]
switch10----|10GbE|switch11[<center>USW-Aggregation<br></center>]
router----|10GbE|switch11[<center>USW-Aggregation<br></center>]



end

subgraph Homelab
router----|1GbE|microshift[<center>MicroShift / PiHole<br>10.0.0.99</center>]
router-.-|1GbE Fail Over|truenas(<center>TrueNas<br>10.101.1.6</center>)
end

subgraph OpenShift/OKD
    switch1---|2.5GbE|truenas(<center>TrueNas<br>10.101.1.6</center>)
    switch1-.-|1GbE Fail Over Host/Pod/LB|server-1(<center>server-1<br>10.101.10.101</center>)
    switch1-.-|1GbE Fail Over Host/Pod/LB|server-2(<center>server-2<br>10.101.10.102</center>)
    switch1-.-|1GbE Fail Over Host/Pod/LB|server-3(<center>server-3<br>10.101.10.103</center>)

    switch1-.-|1GbE Fail Over NAD/LM|server-1(<center>server-1<br>10.101.10.101</center>)
    switch1-.-|1GbE Fail Over NAD/LM|server-2(<center>server-2<br>10.101.10.102</center>)
    switch1-.-|1GbE Fail Over NAD/LM|server-3(<center>server-3<br>10.101.10.103</center>)

    switch1-.-|1GbE Fail Over Ceph Public|server-1(<center>server-1<br>10.101.10.101</center>)
    switch1-.-|1GbE Fail Over Ceph Public|server-2(<center>server-2<br>10.101.10.102</center>)
    switch1-.-|1GbE Fail Over Ceph Public|server-3(<center>server-3<br>10.101.10.103</center>)

    switch1-.-|1GbE Fail Over Ceph Private|server-1(<center>server-1<br>10.101.10.101</center>)
    switch1-.-|1GbE Fail Over Ceph Private|server-2(<center>server-2<br>10.101.10.102</center>)
    switch1-.-|1GbE Fail Over Ceph Private|server-3(<center>server-3<br>10.101.10.103</center>)

    switch10----|10 GbE Host/Pod/LB|server-1(<center>server-1<br>10.101.10.101</center>)
    switch10----|10 GbE Host/Pod/LB|server-2(<center>server-2<br>10.101.10.102</center>)
    switch10----|10 GbE Host/Pod/LB|server-3(<center>server-3<br>10.101.10.103</center>)

    switch10----|10 GbE NAD/LM|server-1(<center>server-1<br>10.101.10.101</center>)
    switch10----|10 GbE NAD/LM|server-2(<center>server-2<br>10.101.10.102</center>)
    switch10----|10 GbE NAD/LM|server-3(<center>server-3<br>10.101.10.103</center>)

    switch11----|10 GbE Ceph Public|server-1(<center>server-1<br>10.101.10.101</center>)
    switch11----|10 GbE Ceph Public|server-2(<center>server-2<br>10.101.10.102</center>)
    switch11----|10 GbE Ceph Public|server-3(<center>server-3<br>10.101.10.103</center>)

    switch11----|10 GbE Ceph Private|server-1(<center>server-1<br>10.101.10.101</center>)
    switch11----|10 GbE Ceph Private|server-2(<center>server-2<br>10.101.10.102</center>)
    switch11----|10 GbE Ceph Private|server-3(<center>server-3<br>10.101.10.103</center>)
end
Loading

Networking

Networking Machines:

NAME Type Status
SB6183 Modem
UDM SE Router
USW Pro Max 16 Switch
USW Aggregation Switch
USW Aggregation Switch
USWFlex XG 10GbE Switch Died
U7 Pro Max AP
U7 Pro Wall AP

Vlans:

Site Range
TODO: 1 10.101.X.X
2 10.102.X.X
Sandbox 10.103.X.X
Name VLAN ID Subnet Info
infra 1 10.102.1.0/24
clients 2 10.102.2.0/24
iot 3 10.102.3.0/24 TODO: Enable Isolation
Name VLAN ID Subnet Info
infrastructure 111 10.101.1.0/24
clients X 10.101.2.0/24 TODO: Create
iot X 10.101.3.0/24 TODO: Create
openshift-legacy 4 10.0.0.0/24 To be Decommissioned
openshift-machine-network 10 10.101.10.0/24 Host/Pod/Load Balancers
openshift-nad 11 10.101.11.0/24 Virtual Machines
openshift-ceph-private 12 10.101.12.0/24 TODO: Enable Isolation
openshift-ceph-public 13 10.101.13.0/24 TODO: Enable Isolation
openshift-lm 14 10.101.14.0/24 TODO: Enable Isolation
openshift-ovn-network 15 10.101.15.0/24 Isolated
openshift-service-network 16 10.101.16.0/22 Isolated
openshift-pod-network 32 10.101.32.0/19 Isolated
Name VLAN ID Subnet Info
virt-openshift-machine-network 310 10.103.10.0/24 Host/Pod/Load Balancers
virt-openshift-nad 311 10.103.11.0/24 Virtual Machines
virt-openshift-ceph-private 312 10.103.12.0/24 TODO: Enable Isolation
virt-openshift-ceph-public 313 10.103.13.0/24 TODO: Enable Isolation
virt-openshift-lm 314 10.103.14.0/24 TODO: Enable Isolation
virt-openshift-ovn-network 315 10.103.15.0/24 Isolated
virt-openshift-service-network 316 10.103.16.0/22 Isolated
virt-openshift-pod-network 332 10.103.32.0/19 Isolated

Kubernetes

https://www.okd.io/

Kubernetes Channel OKD Version OKD Channel OKD OS Storage Layer
v1.33.* 4.19-* stable-scos-4 CentOS Stream CoreOS 9.0 CEPH

Machines

CPU Benchmark

Machine Model CPU CPU Mem Storage Networking ZFS Storage Status
MicroShift Raspberry Pi 5 BCM2712 4 8G 1TB NVME 1x1GbE N/A MicroShift
server-1 N/A R7-5700G 16 128G 2x4TB NVME, 2x1TB SSD, 2x.5TB SSD 4x10Gbe (DAC) && 4x1GbE N/A OpenShift/OKD
server-2 N/A R7-5700G 16 128G 2x4TB NVME, 2x1TB SSD,2x.5TB SSD 4x10Gbe (DAC) && 4x1GbE N/A OpenShift/OKD
server-3 N/A R7-5700G 16 128G 2x4TB NVME, 2x1TB SSD,2x.5TB SSD 4x10Gbe (DAC) && 4x1GbE N/A OpenShift/OKD
TrueNas Hp ProDesk i5-6600 4 32G 120G SSD Boot Mirror 1x2.5Gbe && 1x1GbE 5x2TB RaidZ2 TrueNas
pfSense Hp t730 RX-427BB 4 4G 16G SSD 4x1GbE N/A Decommissioned
Bare Metal Hp t620 GX-415GA 4 6G 16G SSD & 16G USB 1x1GbE N/A Decommissioned
Spare Hp p7-1226s i3-2130 4 8G 240G SSD N/A Decommissioned
Machine PPT CPU Curve GFX Curve CPU Frequency vMem Memory Freq
server-1 40W -20 -30 -750 1.35 3200
server-2 40W -20 -30 -750 1.35 3200
server-3 40W -20 -30 -750 1.35 3200
ZFS Storage
Machine Use Dataset Size Dataset Size Dataset Size Disks (SSD)
TrueNas Backup Nextcloud 750GB Ceph Backup 175GB WindowsBackup 750GB 5x2TB RaidZ2

Kubernetes Nodes

Attribute Value
NAME server-1/2/3
ROLES control-plane, worker
CPU 16
Mem 128G
OS Disk mdadm 2x5TB SSD
Container Overlay mdadm 2x1TB SSD
Ceph Storage 2x4TB CEPH NVME
Nic's 4x10GbE/1GbE A/P
Bond 0: v10 Host/Pod/LoadBalancer
Bond 2: v12 Ceph Private
Bond 3: v13 Ceph Public
Bond 4: v14 VM Live Migrate
Bond 4.3 IOT
Bond 4.11 Virtual Machines
Bond 4.111 Infrastructure
Ceph Performance Tests
test class threads bk-size iops-min iops-max iops-avg MB/s
seq-r cephfs 4 256k 3968 5120 4641 1216
seq-r block 4 256k 3958 5120 4584 1202
seq-w cephfs 4 256k 1024 2048 1661 436
seq-w block 4 256k 1023 2048 1627 427
rand-r cephfs 4 256k 3972 4854 4441 1164
rand-r block 4 256k 3596 4726 4385 1150
rand-w cephfs 4 256k 266 2048 1553 397
rand-w block 4 256k 1402 2048 1617 423
rand-rw cephfs 4 256k 952 1418 1202 313/316
rand-rw block 4 256k 876 1410 1190 310/312

OKD WIF

File Configuration Locations

ls ./terraform/gcp/HomeLab/homelab
ls ./terraform/gcp/HomeLab/homelab/wif
ls ./okd/okd-configuration/wif.yaml

CCOCTL Binary: https://mirror.openshift.com/pub/openshift-v4/amd64/clients/ocp/stable/ccoctl-linux.tar.gz

PROJECT_ID="$(vault kv get -field=project_id secret/gcp/org/av/projects)"

./ccoctl gcp create-workload-identity-pool --name=okd-homelab-wif --project=homelab-${PROJECT_ID} --dry-run
./ccoctl gcp create-workload-identity-provider --name=okd-homelab-wif --region=us --project=homelab-${PROJECT_ID} \
  --public-key-file=serviceaccount-signer.public --workload-identity-pool=okd-homelab-wif --dry-run

Kubernetes Commands

oc login --web --server https://api.okd.homelab.arthurvardevanyan.com:6443

# Watch ALl Pods
watch kubectl get pods -A -o wide --sort-by=.metadata.creationTimestamp
# Delete Pods that Have a Restart
kubectl get pods -A | awk '$5>0' | awk '{print "kubectl delete pod -n " $1 " " $2}' | bash -
# Drain Node
 oc adm drain server-3 --delete-emptydir-data --ignore-daemonsets --force
# Nextcloud
kubectl exec -it nextcloud-0 -n nextcloud -- runuser -u www-data -- php -f /var/www/html/occ
Delete Pod Using Graceful Termination Eviction Request
NAMESPACE=homelab
POD=el-webhook-6b56cc5f84-clfc6

curl --header "Authorization: Bearer $(oc whoami -t)" -H 'Content-type: application/json' \
"$(oc whoami --show-server)/api/v1/namespaces/{$NAMESPACE}/pods/{$POD}/eviction" \
-d '{"apiVersion": "policy/v1","kind": "Eviction","metadata": {"name": "'"${POD}"'","namespace": "'"${NAMESPACE}"'"}}'