Skip to content

Commit 8d1f455

Browse files
authored
Merge pull request #6657 from kameshsampath/registry-aliases
New addon: registry-aliases
2 parents fe3674f + db27b59 commit 8d1f455

File tree

8 files changed

+300
-0
lines changed

8 files changed

+300
-0
lines changed
+149
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Minikube Registry Aliases Addon
2+
3+
An addon to minikube that can help push and pull from the minikube registry using custom domain names. The custom domain names will be made resolveable from with in cluster and at minikube node.
4+
5+
## How to use ?
6+
7+
### Start minikube
8+
9+
```shell
10+
minikube start -p demo
11+
```
12+
This addon depends on `registry` addon, it need to be enabled before the alias addon is installed:
13+
14+
### Enable internal registry
15+
16+
```shell
17+
minikube addons enable registry
18+
```
19+
20+
Verifying the registry deployment
21+
22+
```shell
23+
watch kubectl get pods -n kube-system
24+
```
25+
26+
```shell
27+
NAME READY STATUS RESTARTS AGE
28+
coredns-6955765f44-kpbzt 1/1 Running 0 16m
29+
coredns-6955765f44-lzlsv 1/1 Running 0 16m
30+
etcd-demo 1/1 Running 0 16m
31+
kube-apiserver-demo 1/1 Running 0 16m
32+
kube-controller-manager-demo 1/1 Running 0 16m
33+
kube-proxy-q8rb9 1/1 Running 0 16m
34+
kube-scheduler-demo 1/1 Running 0 16m
35+
*registry-4k8zs* 1/1 Running 0 40s
36+
registry-proxy-vs8jt 1/1 Running 0 40s
37+
storage-provisioner 1/1 Running 0 16m
38+
```
39+
40+
```shell
41+
kubectl get svc -n kube-system
42+
```
43+
44+
```shell
45+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
46+
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 17m
47+
registry ClusterIP 10.97.247.75 <none> 80/TCP 94s
48+
```
49+
50+
>
51+
> **NOTE:**
52+
> Please make a note of the CLUSTER-IP of `registry` service
53+
54+
### Enable registry aliases addon
55+
56+
```shell
57+
minikube addons enable registry-aliases
58+
🌟 The 'registry-aliases' addon is enabled
59+
```
60+
61+
You can check the mikikube vm's `/etc/hosts` file for the registry aliases entries:
62+
63+
```shell
64+
watch minikube ssh -- cat /etc/hosts
65+
```
66+
67+
```shell
68+
127.0.0.1 localhost
69+
127.0.1.1 demo
70+
10.97.247.75 example.org
71+
10.97.247.75 example.com
72+
10.97.247.75 test.com
73+
10.97.247.75 test.org
74+
```
75+
76+
The above output shows that the Daemonset has added the `registryAliases` from the ConfigMap pointing to the internal registry's __CLUSTER-IP__.
77+
78+
### Update CoreDNS
79+
80+
The coreDNS would have been automatically updated by the patch-coredns. A successful job run will have coredns ConfigMap updated like:
81+
82+
```yaml
83+
apiVersion: v1
84+
data:
85+
Corefile: |-
86+
.:53 {
87+
errors
88+
health
89+
rewrite name example.com registry.kube-system.svc.cluster.local
90+
rewrite name example.org registry.kube-system.svc.cluster.local
91+
rewrite name test.com registry.kube-system.svc.cluster.local
92+
rewrite name test.org registry.kube-system.svc.cluster.local
93+
kubernetes cluster.local in-addr.arpa ip6.arpa {
94+
pods insecure
95+
upstream
96+
fallthrough in-addr.arpa ip6.arpa
97+
}
98+
prometheus :9153
99+
proxy . /etc/resolv.conf
100+
cache 30
101+
loop
102+
reload
103+
loadbalance
104+
}
105+
kind: ConfigMap
106+
metadata:
107+
name: coredns
108+
```
109+
110+
To verify it run the following command:
111+
112+
```shell
113+
kubectl get cm -n kube-system coredns -o yaml
114+
```
115+
116+
Once you have successfully patched you can now push and pull from the registry using suffix `example.com`, `example.org`,`test.com` and `test.org`.
117+
118+
The successful run will show the following extra pods (Daemonset, Job) in `kube-system` namespace:
119+
120+
```shell
121+
NAME READY STATUS RESTARTS AGE
122+
registry-aliases-hosts-update-995vx 1/1 Running 0 47s
123+
registry-aliases-patch-core-dns-zsxfc 0/1 Completed 0 47s
124+
```
125+
126+
## Verify with sample application
127+
128+
You can verify the deployment end to end using the example [application](https://github.com/kameshsampath/minikube-registry-aliases-demo).
129+
130+
```shell
131+
git clone https://github.com/kameshsampath/minikube-registry-aliases-demo
132+
cd minikube-registry-aliases-demo
133+
```
134+
135+
Make sure you set the docker context using `eval $(minikube -p demo docker-env)`
136+
137+
Deploy the application using [Skaffold](https://skaffold.dev):
138+
139+
```shell
140+
skaffold dev --port-forward
141+
```
142+
143+
Once the application is running try doing `curl localhost:8080` to see the `Hello World` response
144+
145+
You can also update [skaffold.yaml](./skaffold.yaml) and [app.yaml](.k8s/app.yaml), to use `test.org`, `test.com` or `example.org` as container registry urls, and see all the container image names resolves to internal registry, resulting in successful build and deployment.
146+
147+
> **NOTE**:
148+
>
149+
> You can also update [skaffold.yaml](./skaffold.yaml) and [app. yaml](.k8s/app.yaml), to use `test.org`, `test.com` or > `example.org` as container registry urls, and see all the > container image names resolves to internal registry, resulting in successful build and deployment.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
apiVersion: apps/v1
2+
kind: DaemonSet
3+
metadata:
4+
name: registry-aliases-hosts-update
5+
namespace: kube-system
6+
labels:
7+
kubernetes.io/minikube-addons: registry-aliases
8+
addonmanager.kubernetes.io/mode: Reconcile
9+
spec:
10+
selector:
11+
matchLabels:
12+
app: registry-aliases-hosts-update
13+
template:
14+
metadata:
15+
labels:
16+
app: registry-aliases-hosts-update
17+
spec:
18+
initContainers:
19+
- name: update
20+
image: registry.fedoraproject.org/fedora
21+
volumeMounts:
22+
- name: etchosts
23+
mountPath: /host-etc/hosts
24+
readOnly: false
25+
env:
26+
- name: REGISTRY_ALIASES
27+
valueFrom:
28+
configMapKeyRef:
29+
name: registry-aliases
30+
key: registryAliases
31+
command:
32+
- bash
33+
- -ce
34+
- |
35+
NL=$'\n'
36+
TAB=$'\t'
37+
HOSTS="$(cat /host-etc/hosts)"
38+
[ -z "$REGISTRY_SERVICE_HOST" ] && echo "Failed to get hosts entry for default registry" && exit 1;
39+
for H in $REGISTRY_ALIASES; do
40+
echo "$HOSTS" | grep "$H" || HOSTS="$HOSTS$NL$REGISTRY_SERVICE_HOST$TAB$H";
41+
done;
42+
echo "$HOSTS" | diff -u /host-etc/hosts - || echo "$HOSTS" > /host-etc/hosts
43+
echo "Done."
44+
containers:
45+
- name: pause-for-update
46+
image: gcr.io/google_containers/pause-amd64:3.1
47+
terminationGracePeriodSeconds: 30
48+
volumes:
49+
- name: etchosts
50+
hostPath:
51+
path: /etc/hosts
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
apiVersion: batch/v1
3+
kind: Job
4+
metadata:
5+
name: registry-aliases-patch-core-dns
6+
namespace: kube-system
7+
spec:
8+
ttlSecondsAfterFinished: 100
9+
template:
10+
spec:
11+
serviceAccountName: registry-aliases-sa
12+
volumes:
13+
- name: minikube
14+
hostPath:
15+
path: /var/lib/minikube/binaries
16+
containers:
17+
- name: core-dns-patcher
18+
image: quay.io/rhdevelopers/core-dns-patcher
19+
imagePullPolicy: IfNotPresent
20+
# using the kubectl from the minikube instance
21+
volumeMounts:
22+
- mountPath: /var/lib/minikube/binaries
23+
name: minikube
24+
readOnly: true
25+
restartPolicy: Never
26+
backoffLimit: 4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: registry-aliases
5+
namespace: kube-system
6+
labels:
7+
kubernetes.io/minikube-addons: registry-aliases
8+
addonmanager.kubernetes.io/mode: Reconcile
9+
data:
10+
# Add additonal hosts seperated by new-line
11+
registryAliases: >-
12+
example.org
13+
example.com
14+
test.com
15+
test.org
16+
# default registry address in minikube when enabled via minikube addons enable registry
17+
registrySvc: registry.kube-system.svc.cluster.local
18+
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: rbac.authorization.k8s.io/v1
2+
kind: ClusterRoleBinding
3+
metadata:
4+
name: registry-aliases-crb
5+
subjects:
6+
- kind: ServiceAccount
7+
name: registry-aliases-sa
8+
namespace: kube-system
9+
roleRef:
10+
kind: ClusterRole
11+
name: cluster-admin
12+
apiGroup: rbac.authorization.k8s.io
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
apiVersion: v1
2+
kind: ServiceAccount
3+
metadata:
4+
name: registry-aliases-sa
5+
namespace: kube-system

pkg/addons/config.go

+7
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,13 @@ var Addons = []*Addon{
112112
set: SetBool,
113113
callbacks: []setFn{enableOrDisableAddon},
114114
},
115+
{
116+
name: "registry-aliases",
117+
set: SetBool,
118+
callbacks: []setFn{enableOrDisableAddon},
119+
//TODO - add other settings
120+
//TODO check if registry addon is enabled
121+
},
115122
{
116123
name: "storage-provisioner",
117124
set: SetBool,

pkg/minikube/assets/addons.go

+32
Original file line numberDiff line numberDiff line change
@@ -246,6 +246,38 @@ var Addons = map[string]*Addon{
246246
"0640",
247247
false),
248248
}, false, "registry-creds"),
249+
"registry-aliases": NewAddon([]*BinAsset{
250+
MustBinAsset(
251+
"deploy/addons/registry-aliases/registry-aliases-sa.tmpl",
252+
vmpath.GuestAddonsDir,
253+
"registry-aliases-sa.yaml",
254+
"0640",
255+
false),
256+
MustBinAsset(
257+
"deploy/addons/registry-aliases/registry-aliases-sa-crb.tmpl",
258+
vmpath.GuestAddonsDir,
259+
"registry-aliases-sa-crb.yaml",
260+
"0640",
261+
false),
262+
MustBinAsset(
263+
"deploy/addons/registry-aliases/registry-aliases-config.tmpl",
264+
vmpath.GuestAddonsDir,
265+
"registry-aliases-config.yaml",
266+
"0640",
267+
false),
268+
MustBinAsset(
269+
"deploy/addons/registry-aliases/node-etc-hosts-update.tmpl",
270+
vmpath.GuestAddonsDir,
271+
"node-etc-hosts-update.yaml",
272+
"0640",
273+
false),
274+
MustBinAsset(
275+
"deploy/addons/registry-aliases/patch-coredns-job.tmpl",
276+
vmpath.GuestAddonsDir,
277+
"patch-coredns-job.yaml",
278+
"0640",
279+
false),
280+
}, false, "registry-aliases"),
249281
"freshpod": NewAddon([]*BinAsset{
250282
MustBinAsset(
251283
"deploy/addons/freshpod/freshpod-rc.yaml.tmpl",

0 commit comments

Comments
 (0)