-
Notifications
You must be signed in to change notification settings - Fork 54
Backport docker examples for 2018 #99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
b8fc8cd
Adds docker-compose support for testing purposes
logicminds e8db249
Add ability to audit the files being copied over
logicminds c607e50
Use dynamic directories with bolt in example docker
logicminds 84e88ee
Update context of docker files
logicminds 9da423b
Add puppetdb.conf to docker containers
logicminds 4b76740
Fix locale issues
logicminds a1c29e6
Always restart docker container
logicminds e35d781
Update parameters to use 2018 stack
logicminds 141b627
Make cert tasks more idempotent
logicminds File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
--- | ||
fixtures: | ||
repositories: | ||
facts: 'https://github.com/puppetlabs/puppetlabs-facts.git' | ||
puppet_agent: 'https://github.com/puppetlabs/puppetlabs-puppet_agent.git' | ||
provision: 'https://github.com/puppetlabs/provision.git' | ||
stdlib: 'https://github.com/puppetlabs/puppetlabs-stdlib.git' | ||
node_manager: 'https://github.com/WhatsARanjit/puppet-node_manager' | ||
apply_helpers: 'https://github.com/puppetlabs/puppetlabs-apply_helpers' | ||
bolt_shim: 'https://github.com/puppetlabs/puppetlabs-bolt_shim' | ||
debug: 'https://github.com/nwops/puppet-debug' | ||
symlinks: | ||
"pe_xl": "#{source_dir}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -25,3 +25,5 @@ | |
.project | ||
.envrc | ||
/inventory.yaml | ||
.rerun.json | ||
*.tar.gz |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
2.6.5 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
## Docker Based Examples | ||
This module provides docker compose files for the various architectures for experimentation purposes. This gives you the ability to stand up an entire PE stack in order to learn how this module and HA works. If you have docker and docker-compose you can start up a full Puppet architecture with a single command. Please note that Puppet does not support PE on containers in production. | ||
|
||
In order to decouple Bolt from a dev system, a special bolt container is created that will run all the bolt commands. This is | ||
required to achieve maximum portability. Should you want to run bolt commands against the PE stack you must | ||
first login to this bolt container via ssh, docker or docker-compose. | ||
|
||
Example: `docker-compose run --entrypoint=/bin/bash bolt` | ||
|
||
### Requirements | ||
To run the container based examples you will need the following requirements: | ||
|
||
2. Docker | ||
3. Docker compose | ||
4. 16GB memory, 24GB+ for XL and XL-HA architectures | ||
5. CPU with many cores (Tested with Core i7 6700) | ||
|
||
### Starting the example | ||
We have provided a provision.sh script to help making these examples simple. | ||
To use perform the following: | ||
|
||
1. cd spec/docker | ||
2. bash provision.sh | ||
3. select desired architecture when prompted (ie. extra-large-ha ) | ||
4. Wait 10-20 minutes for provisioning to complete | ||
|
||
``` | ||
Please choose a PE architecture to build: | ||
1) extra-large/ 3) large/ 5) standard/ | ||
2) extra-large-ha/ 4) large-ha/ 6) standard-ha/ | ||
#? | ||
``` | ||
|
||
### Stopping the example | ||
In order to stop and remove the containers you will need to perform the following. | ||
|
||
1. cd spec/docker | ||
2. `cd <chosen architecture>` | ||
3. docker-compose down | ||
|
||
### Logging into the console | ||
You can login to the PE Console after successful provision. However, first you will need to | ||
grab the mapped port number of the PE console. The port numbers are mapped dynamically as to not | ||
cause port conflicts on your system. To see how the ports are mapped you can view them via: | ||
|
||
1. docker ps | ||
``` | ||
80c6f0b5525c pe-base "/sbin/init" 2 hours ago Up 2 hours 0.0.0.0:32774->22/tcp, 0.0.0.0:32773->443/tcp, 0.0.0.0:32772->4433/tcp, 0.0.0.0:32771->8080/tcp, 0.0.0.0:32770->8081/tcp, 0.0.0.0:32769->8140/tcp, 0.0.0.0:32768->8443/tcp pe-lg.puppet.vm | ||
``` | ||
2. Note the mapped port for 443, which in this case is 32773 | ||
3. Visit https://localhost:32773 in your browser | ||
4. Accept security risk (self signed cert) | ||
5. Login: admin/puppetlabs | ||
|
||
### Logging into any of the containers | ||
Ssh is running in all the containers so you can use ssh if you grab the mapped ssh port number. `ssh root@localhost -p 32774` | ||
|
||
Login: root/test | ||
|
||
You can also bypass ssh and run docker exec or docker-compose exec | ||
|
||
1. cd spec/docker/extra-large | ||
2. docker-compose exec pe_xl_core /bin/bash | ||
|
||
**Note:** pe_xl_core is the name of the service defined in the respective docker-compose file. | ||
|
||
This will run an interactive bash shell in the running container. | ||
|
||
### Upgrades | ||
There is also a upgrade.sh script that is similar to the provision.sh script. This script will upgrade an already provisioned PE stack to the version specified in the update_params.json file. | ||
|
||
### Other notes | ||
1. The provision plan is not fully idempotent. | ||
2. Some tasks may fail when run due to resource constraints. | ||
3. You can rerun the provision.sh script on the same architecture without destroying the containers. This can sometimes complete the provision process successfully. | ||
4. Rerunning the provision script may result in errors due to idempotency issues with tasks and plans. | ||
5. Please remember you are starting the equilivent of 3-6 VMs on a single system. | ||
6. You can use top to view all the processes being run in the containers. | ||
7. Docker will use the privilege mode option when running these examples (systemd support) | ||
8. Systemd is running inside these containers! The real systemd, not the fake one. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
*.tar.gz |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
# In order to run this with systemd you must do one of the following | ||
# use --privileged when running the container | ||
# mount the volume when running the container -v /sys/fs/cgroup:/sys/fs/cgroup:ro | ||
# docker run –privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 2222:22 | ||
FROM centos:7 | ||
EXPOSE 22 | ||
ENV LC_ALL="en_US.UTF-8" LANG="en_US.UTF-8" LANGUAGE="en_US.UTF-8" | ||
RUN echo "LANG=en_US.UTF-8" > /etc/locale.conf | ||
STOPSIGNAL SIGRTMIN+3 | ||
ADD live_audit.sh /usr/bin/live_audit.sh | ||
ADD live_audit.service /etc/systemd/system/live_audit.service | ||
RUN chmod 644 /etc/systemd/system/live_audit.service && chmod 755 /usr/bin/live_audit.sh && \ | ||
echo "root:test" | chpasswd; \ | ||
useradd -m -s /bin/bash centos && echo "centos:test" | chpasswd; | ||
RUN yum -y install epel-release systemd rsync tree vim openssh openssh-server openssh-clients anacron sudo curl openssl | ||
RUN yum -y install inotify-tools && mkdir /root/bolt_scripts && yum clean all | ||
# remove any scripts that don't need to be run | ||
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \ | ||
rm -f /lib/systemd/system/multi-user.target.wants/*; \ | ||
rm -f /etc/systemd/system/*.wants/*; \ | ||
rm -f /lib/systemd/system/local-fs.target.wants/*; \ | ||
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \ | ||
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \ | ||
rm -f /lib/systemd/system/basic.target.wants/*; \ | ||
rm -f /lib/systemd/system/anaconda.target.wants/*; \ | ||
rm -rf /var/cache/yum; \ | ||
ln -s '/usr/lib/systemd/system/sshd.service' '/etc/systemd/system/multi-user.target.wants/sshd.service' && \ | ||
ln -s '/etc/systemd/system/live_audit.service' '/etc/systemd/system/multi-user.target.wants/live_audit.service' | ||
ARG HOST="localhost" | ||
RUN mkdir -p /root/.puppetlabs/client-tools && \ | ||
echo $' \n\ | ||
{ \n\ | ||
"puppetdb": { \n\ | ||
"server_urls": "https://'$HOST$':8081", \n\ | ||
"cacert": "/etc/puppetlabs/puppet/ssl/certs/ca.pem", \n\ | ||
"cert": "/etc/puppetlabs/puppet/ssl/certs/'$HOST$'.pem", \n\ | ||
"key": "/etc/puppetlabs/puppet/ssl/private_keys/'$HOST$'.pem" \n\ | ||
} \n\ | ||
} \n '\ | ||
> /root/.puppetlabs/client-tools/puppetdb.conf | ||
#VOLUME [ “/sys/fs/cgroup” ] | ||
#CMD /bin/bash | ||
ENTRYPOINT [ "/sbin/init" ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
FROM puppet/puppet-bolt | ||
RUN apt-get update && apt-get install -y locales ssh sudo curl; \ | ||
/opt/puppetlabs/bolt/bin/gem install bundler puppet-debugger -N -q | ||
ENV LC_ALL="en_US.UTF-8" LANG="en_US.UTF-8" LANGUAGE="en_US.UTF-8" | ||
RUN echo "LC_ALL=en_US.UTF-8" >> /etc/environment && \ | ||
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen && \ | ||
echo "LANG=en_US.UTF-8" > /etc/locale.conf && \ | ||
locale-gen en_US.UTF-8 | ||
CMD /bin/bash | ||
ENTRYPOINT [ "/opt/puppetlabs/bin/bolt" ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,129 @@ | ||
version: "3" | ||
services: | ||
bolt: | ||
build: | ||
dockerfile: 'Dockerfile_bolt' | ||
context: ../ | ||
image: pe-bolt | ||
hostname: bolter.puppet.vm | ||
container_name: bolter.puppet.vm | ||
volumes: | ||
- .:/app | ||
working_dir: /app | ||
depends_on: | ||
- pe_xl_core_0 | ||
compiler1: | ||
restart: always | ||
depends_on: | ||
- pe_xl_core_0 | ||
build: | ||
dockerfile: 'Dockerfile' | ||
context: ../ | ||
entrypoint: /sbin/init | ||
image: pe-base | ||
privileged: true # required for systemd | ||
ports: | ||
- "22" | ||
- "8140" | ||
hostname: pe-xl-compiler-0.puppet.vm | ||
container_name: pe-xl-compiler-0.puppet.vm | ||
stop_signal: SIGRTMIN+3 | ||
tmpfs: | ||
- /run | ||
- /tmp | ||
volumes: | ||
- '/sys/fs/cgroup:/sys/fs/cgroup:ro' | ||
pe_pdb: | ||
restart: always | ||
depends_on: | ||
- pe_xl_core_0 | ||
build: | ||
dockerfile: 'Dockerfile' | ||
context: ../ | ||
entrypoint: /sbin/init | ||
image: pe-base | ||
privileged: true # required for systemd | ||
ports: | ||
- "22" | ||
hostname: pe-xl-db-0.puppet.vm | ||
container_name: pe-xl-db-0.puppet.vm | ||
stop_signal: SIGRTMIN+3 | ||
tmpfs: | ||
- /run | ||
- /tmp | ||
volumes: | ||
- '/sys/fs/cgroup:/sys/fs/cgroup:ro' | ||
pe_pdb-replica: | ||
restart: always | ||
depends_on: | ||
- pe_xl_core_0 | ||
build: | ||
dockerfile: 'Dockerfile' | ||
context: ../ | ||
entrypoint: /sbin/init | ||
image: pe-base | ||
privileged: true # required for systemd | ||
ports: | ||
- "22" | ||
hostname: pe-xl-db-1.puppet.vm | ||
container_name: pe-xl-db-1.puppet.vm | ||
stop_signal: SIGRTMIN+3 | ||
tmpfs: | ||
- /run | ||
- /tmp | ||
volumes: | ||
- '/sys/fs/cgroup:/sys/fs/cgroup:ro' | ||
pe_xl_core_1: | ||
restart: always | ||
depends_on: | ||
- pe_xl_core_0 | ||
build: | ||
dockerfile: 'Dockerfile' | ||
context: ../ | ||
args: | ||
HOST: 'pe-xl-core-1.puppet.vm' | ||
entrypoint: /sbin/init | ||
image: pe-base | ||
privileged: true # required for systemd | ||
ports: | ||
- "22" | ||
- "8140" | ||
- "8443" | ||
- "8080" | ||
- "8081" | ||
- "4433" | ||
- "443" | ||
container_name: pe-xl-core-1.puppet.vm | ||
hostname: pe-xl-core-1.puppet.vm | ||
stop_signal: SIGRTMIN+3 | ||
tmpfs: | ||
- /run | ||
- /tmp | ||
volumes: | ||
- '/sys/fs/cgroup:/sys/fs/cgroup:ro' | ||
pe_xl_core_0: | ||
restart: always | ||
build: | ||
dockerfile: 'Dockerfile' | ||
context: ../ | ||
args: | ||
HOST: 'pe-xl-core-0.puppet.vm' | ||
entrypoint: /sbin/init | ||
image: pe-base | ||
privileged: true # required for systemd | ||
ports: | ||
- "22" | ||
- "8140" | ||
- "8443" | ||
- "8080" | ||
- "8081" | ||
- "4433" | ||
- "443" | ||
hostname: pe-xl-core-0.puppet.vm | ||
container_name: pe-xl-core-0.puppet.vm | ||
stop_signal: SIGRTMIN+3 | ||
tmpfs: | ||
- /run | ||
- /tmp | ||
volumes: | ||
- '/sys/fs/cgroup:/sys/fs/cgroup:ro' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
--- | ||
groups: | ||
- name: puppet-enterprise-nodes | ||
config: | ||
transport: ssh | ||
ssh: | ||
tmpdir: /root/bolt_scripts | ||
host-key-check: false | ||
user: root | ||
password: test | ||
tty: true | ||
targets: | ||
- pe-xl-compiler-0.puppet.vm | ||
- pe-xl-db-0.puppet.vm | ||
- pe-xl-db-1.puppet.vm | ||
- pe-xl-core-0.puppet.vm | ||
- pe-xl-core-1.puppet.vm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
{ | ||
"master_host": "pe-xl-core-0.puppet.vm", | ||
"puppetdb_database_host": "pe-xl-db-0.puppet.vm", | ||
"puppetdb_database_replica_host": "pe-xl-db-1.puppet.vm", | ||
"master_replica_host": "pe-xl-core-1.puppet.vm", | ||
"compiler_hosts": ["pe-xl-compiler-0.puppet.vm"], | ||
"console_password": "puppetlabs", | ||
"dns_alt_names": [ "puppet", "pe-xl-core-0.puppet.vm", "puppet-xl.vm" ], | ||
"version": "2018.1.13", | ||
"compiler_pool_address": "puppet-xl.vm", | ||
"stagingdir": "/downloads" | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
{ | ||
"master_host": "pe-xl-core-0.puppet.vm", | ||
"puppetdb_database_host": "pe-xl-db-0.puppet.vm", | ||
"puppetdb_database_replica_host": "pe-xl-db-1.puppet.vm", | ||
"master_replica_host": "pe-xl-core-1.puppet.vm", | ||
"compiler_hosts": ["pe-xl-compiler-0.puppet.vm"], | ||
"version": "2018.1.15", | ||
"stagingdir": "/downloads" | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Below is where a bug was introduced when agent_installer_targets was empty. Now fixed with the new task and conditional around the task.