Skip to content

Commit f794392

Browse files
committed
Adds docker-compose support for testing purposes
* Previously working on this module required a buildout of a puppet infrastructure with VMs or cloud providers. Which can sometimes be costly or take hours to deploy. This commit uses containers and docker-compose to produce sample environments for each architecuture in order to perform rapid prototyping or test the functionality of this module and the puppet infra command. The architectures supported in this command are: * standard, stadard-ha * large, large-ha * extra-large, extra-large-ha While there are many other possiblites and combinations of amount of servers and containers these are the current choices. For more information please see documention/docker_examples.md
1 parent 6e59ab9 commit f794392

33 files changed

+802
-1
lines changed

.fixtures.yml

+1
Original file line numberDiff line numberDiff line change
@@ -8,5 +8,6 @@ fixtures:
88
node_manager: 'https://github.com/WhatsARanjit/puppet-node_manager'
99
apply_helpers: 'https://github.com/puppetlabs/puppetlabs-apply_helpers'
1010
bolt_shim: 'https://github.com/puppetlabs/puppetlabs-bolt_shim'
11+
debug: 'https://github.com/nwops/puppet-debug'
1112
symlinks:
1213
"peadm": "#{source_dir}"

.gitignore

+2-1
Original file line numberDiff line numberDiff line change
@@ -25,4 +25,5 @@
2525
.project
2626
.envrc
2727
/inventory.yaml
28-
.rerun.json
28+
.rerun.json
29+
*.tar.gz

.ruby-version

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
2.6.5

README.md

+1
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,4 @@ Reference:
2323
* [Classification](documentation/classification.md)
2424
* [Architectures](documentation/architectures.md)
2525
* [Testing](documentation/pre_post_checks.md)
26+
* [Docker Based Examples](documentation/docker_examples.md)

documentation/docker_examples.md

+80
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
## Docker Based Examples
2+
This module provides docker compose files for the various architectures for experimentation purposes. This gives you the ability to stand up an entire PE stack in order to learn how this module and HA works. If you have docker and docker-compose you can start up a full Puppet architecture with a single command. Please note that Puppet does not support PE on containers in production.
3+
4+
In order to decouple Bolt from a dev system, a special bolt container is created that will run all the bolt commands. This is
5+
required to achieve maximum portability. Should you want to run bolt commands against the PE stack you must
6+
first login to this bolt container via ssh, docker or docker-compose.
7+
8+
Example: `docker-compose run --entrypoint=/bin/bash bolt`
9+
10+
### Requirements
11+
To run the container based examples you will need the following requirements:
12+
13+
2. Docker
14+
3. Docker compose
15+
4. 16GB memory, 24GB+ for XL and XL-HA architectures
16+
5. CPU with many cores (Tested with Core i7 6700)
17+
18+
### Starting the example
19+
We have provided a provision.sh script to help making these examples simple.
20+
To use perform the following:
21+
22+
1. cd spec/docker
23+
2. bash provision.sh
24+
3. select desired architecture when prompted (ie. extra-large-ha )
25+
4. Wait 10-20 minutes for provisioning to complete
26+
27+
```
28+
Please choose a PE architecture to build:
29+
1) extra-large/ 3) large/ 5) standard/
30+
2) extra-large-ha/ 4) large-ha/ 6) standard-ha/
31+
#?
32+
```
33+
34+
### Stopping the example
35+
In order to stop and remove the containers you will need to perform the following.
36+
37+
1. cd spec/docker
38+
2. `cd <chosen architecture>`
39+
3. docker-compose down
40+
41+
### Logging into the console
42+
You can login to the PE Console after successful provision. However, first you will need to
43+
grab the mapped port number of the PE console. The port numbers are mapped dynamically as to not
44+
cause port conflicts on your system. To see how the ports are mapped you can view them via:
45+
46+
1. docker ps
47+
```
48+
80c6f0b5525c pe-base "/sbin/init" 2 hours ago Up 2 hours 0.0.0.0:32774->22/tcp, 0.0.0.0:32773->443/tcp, 0.0.0.0:32772->4433/tcp, 0.0.0.0:32771->8080/tcp, 0.0.0.0:32770->8081/tcp, 0.0.0.0:32769->8140/tcp, 0.0.0.0:32768->8443/tcp pe-lg.puppet.vm
49+
```
50+
2. Note the mapped port for 443, which in this case is 32773
51+
3. Visit https://localhost:32773 in your browser
52+
4. Accept security risk (self signed cert)
53+
5. Login: admin/puppetlabs
54+
55+
### Logging into any of the containers
56+
Ssh is running in all the containers so you can use ssh if you grab the mapped ssh port number. `ssh root@localhost -p 32774`
57+
58+
Login: root/test
59+
60+
You can also bypass ssh and run docker exec or docker-compose exec
61+
62+
1. cd spec/docker/extra-large
63+
2. docker-compose exec pe_xl_core /bin/bash
64+
65+
**Note:** pe_xl_core is the name of the service defined in the respective docker-compose file.
66+
67+
This will run an interactive bash shell in the running container.
68+
69+
### Upgrades
70+
There is also a upgrade.sh script that is similar to the provision.sh script. This script will upgrade an already provisioned PE stack to the version specified in the update_params.json file.
71+
72+
### Other notes
73+
1. The provision plan is not fully idempotent.
74+
2. Some tasks may fail when run due to resource constraints.
75+
3. You can rerun the provision.sh script on the same architecture without destroying the containers. This can sometimes complete the provision process successfully.
76+
4. Rerunning the provision script may result in errors due to idempotency issues with tasks and plans.
77+
5. Please remember you are starting the equilivent of 3-6 VMs on a single system.
78+
6. You can use top to view all the processes being run in the containers.
79+
7. Docker will use the privilege mode option when running these examples (systemd support)
80+
8. Systemd is running inside these containers! The real systemd, not the fake one.

spec/docker/Dockerfile

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# In order to run this with systemd you must do one of the following
2+
# use --privileged when running the container
3+
# mount the volume when running the container -v /sys/fs/cgroup:/sys/fs/cgroup:ro
4+
# docker run –privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 2222:22
5+
FROM centos:7
6+
EXPOSE 22
7+
ENV LC_ALL="en_US.UTF-8" LANG="en_US.UTF-8" LANGUAGE="en_US.UTF-8"
8+
RUN echo "LANG=en_US.UTF-8" > /etc/locale.conf
9+
STOPSIGNAL SIGRTMIN+3
10+
RUN yum -y install systemd openssh openssh-server openssh-clients anacron sudo curl openssl; yum clean all;
11+
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \
12+
rm -f /lib/systemd/system/multi-user.target.wants/*; \
13+
rm -f /etc/systemd/system/*.wants/*; \
14+
rm -f /lib/systemd/system/local-fs.target.wants/*; \
15+
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
16+
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
17+
rm -f /lib/systemd/system/basic.target.wants/*; \
18+
rm -f /lib/systemd/system/anaconda.target.wants/*; \
19+
rm -rf /var/cache/yum; \
20+
echo "root:test" | chpasswd; \
21+
useradd -m -s /bin/bash centos && echo "centos:test" | chpasswd; \
22+
ln -s '/usr/lib/systemd/system/sshd.service' '/etc/systemd/system/multi-user.target.wants/sshd.service'
23+
VOLUME [ “/sys/fs/cgroup” ]
24+
CMD /sbin/init
25+
# ENTRYPOINT [ "/sbin/init" ]

spec/docker/Dockerfile_bolt

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
FROM puppet/puppet-bolt
2+
ENV LC_ALL="en_US.UTF-8" LANG="en_US.UTF-8" LANGUAGE="en_US.UTF-8"
3+
RUN echo "LANG=en_US.UTF-8" > /etc/locale.conf
4+
RUN apt-get update && apt-get install -y ssh sudo curl; \
5+
/opt/puppetlabs/bolt/bin/gem install bundler puppet-debugger -N -q
6+
CMD /bin/bash
7+
ENTRYPOINT [ "/opt/puppetlabs/bin/bolt" ]
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
version: "3"
2+
services:
3+
bolt:
4+
build:
5+
dockerfile: '../Dockerfile_bolt'
6+
context: .
7+
image: pe-bolt
8+
hostname: bolter.puppet.vm
9+
container_name: bolter.puppet.vm
10+
volumes:
11+
- .:/app
12+
working_dir: /app
13+
depends_on:
14+
- pe_xl_core_0
15+
compiler1:
16+
depends_on:
17+
- pe_xl_core_0
18+
build:
19+
dockerfile: '../Dockerfile'
20+
context: .
21+
entrypoint: /sbin/init
22+
image: pe-base
23+
privileged: true # required for systemd
24+
ports:
25+
- "22"
26+
- "8140"
27+
hostname: pe-xl-compiler-0.puppet.vm
28+
container_name: pe-xl-compiler-0.puppet.vm
29+
stop_signal: SIGRTMIN+3
30+
tmpfs:
31+
- /run
32+
- /tmp
33+
volumes:
34+
- '/sys/fs/cgroup:/sys/fs/cgroup:ro'
35+
pe_pdb:
36+
depends_on:
37+
- pe_xl_core_0
38+
build:
39+
dockerfile: '../Dockerfile'
40+
context: .
41+
entrypoint: /sbin/init
42+
image: pe-base
43+
privileged: true # required for systemd
44+
ports:
45+
- "22"
46+
hostname: pe-xl-db-0.puppet.vm
47+
container_name: pe-xl-db-0.puppet.vm
48+
stop_signal: SIGRTMIN+3
49+
tmpfs:
50+
- /run
51+
- /tmp
52+
volumes:
53+
- '/sys/fs/cgroup:/sys/fs/cgroup:ro'
54+
pe_pdb-replica:
55+
depends_on:
56+
- pe_xl_core_0
57+
build:
58+
dockerfile: '../Dockerfile'
59+
context: .
60+
entrypoint: /sbin/init
61+
image: pe-base
62+
privileged: true # required for systemd
63+
ports:
64+
- "22"
65+
hostname: pe-xl-db-1.puppet.vm
66+
container_name: pe-xl-db-1.puppet.vm
67+
stop_signal: SIGRTMIN+3
68+
tmpfs:
69+
- /run
70+
- /tmp
71+
volumes:
72+
- '/sys/fs/cgroup:/sys/fs/cgroup:ro'
73+
pe_xl_core_1:
74+
depends_on:
75+
- pe_xl_core_0
76+
build:
77+
dockerfile: '../Dockerfile'
78+
context: .
79+
entrypoint: /sbin/init
80+
image: pe-base
81+
privileged: true # required for systemd
82+
ports:
83+
- "22"
84+
- "8140"
85+
- "8443"
86+
- "8080"
87+
- "8081"
88+
- "4433"
89+
- "443"
90+
container_name: pe-xl-core-1.puppet.vm
91+
hostname: pe-xl-core-1.puppet.vm
92+
stop_signal: SIGRTMIN+3
93+
tmpfs:
94+
- /run
95+
- /tmp
96+
volumes:
97+
- '/sys/fs/cgroup:/sys/fs/cgroup:ro'
98+
pe_xl_core_0:
99+
build:
100+
dockerfile: '../Dockerfile'
101+
context: .
102+
entrypoint: /sbin/init
103+
image: pe-base
104+
privileged: true # required for systemd
105+
ports:
106+
- "22"
107+
- "8140"
108+
- "8443"
109+
- "8080"
110+
- "8081"
111+
- "4433"
112+
- "443"
113+
hostname: pe-xl-core-0.puppet.vm
114+
container_name: pe-xl-core-0.puppet.vm
115+
stop_signal: SIGRTMIN+3
116+
tmpfs:
117+
- /run
118+
- /tmp
119+
volumes:
120+
- '/sys/fs/cgroup:/sys/fs/cgroup:ro'
+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
groups:
3+
- name: puppet-enterprise-nodes
4+
config:
5+
transport: ssh
6+
ssh:
7+
tmpdir: /root
8+
script-dir: test123
9+
host-key-check: false
10+
user: root
11+
password: test
12+
tty: true
13+
targets:
14+
- pe-xl-compiler-0.puppet.vm
15+
- pe-xl-db-0.puppet.vm
16+
- pe-xl-db-1.puppet.vm
17+
- pe-xl-core-0.puppet.vm
18+
- pe-xl-core-1.puppet.vm
+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"master_host": "pe-xl-core-0.puppet.vm",
3+
"puppetdb_database_host": "pe-xl-db-0.puppet.vm",
4+
"puppetdb_database_replica_host": "pe-xl-db-1.puppet.vm",
5+
"master_replica_host": "pe-xl-core-1.puppet.vm",
6+
"compiler_hosts": ["pe-xl-compiler-0.puppet.vm"],
7+
"console_password": "puppetlabs",
8+
"dns_alt_names": [ "puppet", "pe-xl-core-0.puppet.vm", "puppet-xl.vm" ],
9+
"version": "2019.1.1",
10+
"compiler_pool_address": "puppet-xl.vm",
11+
"stagingdir": "/downloads"
12+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"master_host": "pe-xl-core-0.puppet.vm",
3+
"puppetdb_database_host": "pe-xl-db-0.puppet.vm",
4+
"puppetdb_database_replica_host": "pe-xl-db-1.puppet.vm",
5+
"master_replica_host": "pe-xl-core-1.puppet.vm",
6+
"compiler_hosts": ["pe-xl-compiler-0.puppet.vm"],
7+
"version": "2019.5.0",
8+
"stagingdir": "/downloads"
9+
}
10+
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
version: "3"
2+
services:
3+
bolt:
4+
build:
5+
dockerfile: '../Dockerfile_bolt'
6+
context: .
7+
image: pe-build
8+
hostname: bolter.puppet.vm
9+
container_name: bolter.puppet.vm
10+
volumes:
11+
- .:/app
12+
working_dir: /app
13+
depends_on:
14+
- pe_xl_core
15+
compiler1:
16+
depends_on:
17+
- pe_xl_core
18+
build:
19+
dockerfile: '../Dockerfile'
20+
context: .
21+
entrypoint: /sbin/init
22+
image: pe-base
23+
privileged: true # required for systemd
24+
ports:
25+
- "22"
26+
- "8140"
27+
hostname: pe-xl-compiler-0.puppet.vm
28+
container_name: pe-xl-compiler-0.puppet.vm
29+
stop_signal: SIGRTMIN+3
30+
tmpfs:
31+
- /run
32+
- /tmp
33+
volumes:
34+
- '/sys/fs/cgroup:/sys/fs/cgroup:ro'
35+
pe_pdb:
36+
depends_on:
37+
- pe_xl_core
38+
build:
39+
dockerfile: '../Dockerfile'
40+
context: .
41+
entrypoint: /sbin/init
42+
image: pe-base
43+
privileged: true # required for systemd
44+
ports:
45+
- "22"
46+
hostname: pe-xl-db-0.puppet.vm
47+
container_name: pe-xl-db-0.puppet.vm
48+
stop_signal: SIGRTMIN+3
49+
tmpfs:
50+
- /run
51+
- /tmp
52+
volumes:
53+
- '/sys/fs/cgroup:/sys/fs/cgroup:ro'
54+
pe_xl_core:
55+
build:
56+
dockerfile: '../Dockerfile'
57+
context: .
58+
entrypoint: /sbin/init
59+
image: pe-base
60+
privileged: true # required for systemd
61+
ports:
62+
- "22"
63+
- "8140"
64+
- "8443"
65+
- "8080"
66+
- "8081"
67+
- "4433"
68+
- "443"
69+
hostname: pe-xl-core-0.puppet.vm
70+
container_name: pe-xl-core-0.puppet.vm
71+
stop_signal: SIGRTMIN+3
72+
tmpfs:
73+
- /run
74+
- /tmp
75+
volumes:
76+
- '/sys/fs/cgroup:/sys/fs/cgroup:ro'
+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
groups:
3+
- name: puppet-enterprise-nodes
4+
config:
5+
transport: ssh
6+
ssh:
7+
tmpdir: /root
8+
script-dir: test123
9+
host-key-check: false
10+
user: root
11+
password: test
12+
tty: true
13+
targets:
14+
- pe-xl-compiler-0.puppet.vm
15+
- pe-xl-db-0.puppet.vm
16+
- pe-xl-core-0.puppet.vm

0 commit comments

Comments
 (0)