Skip to content

Commit 68329f0

Browse files
CoMfUcIoSJo-Lillieragingracathal41timidri
authored
(PE-37233-35) Official support of Add compiler plan (#485)
* (PE-37236) Remove experimental status of add_compiler (#428) * (PE-37233) Adding add_compiler to test matrix (#434) Updated add_compiler spec to check compiler has been added and puppet run before running add compiler Added test matrix for add compiler * (PE-38814) add_compiler - Making primary_postgresql_host and avail_group_letter optional (#468) * (PE-38814) add_compiler - Making primary_postgresql_host and avail_group_letter optional primary_postgresql_host, if not provided will be determined through get_peadm_config avail_group_letter, is defaulting to A * Updating spec tests for add compiler Removing optional from avail_group_letter as not required with enum default value * Updating reference.md * Fixing linting issues --------- Co-authored-by: Neil Anderson <[email protected]> * (PE-37235) Verify and update documentation on add_compiler (#429) * PE-37235 Verify and update documentation on add_compiler if needed and its experimental status * @cathal41 PE-37235 Verify and update documentation on add_compiler - wording updates * Update plans/add_compiler.pp Co-authored-by: Dimitri Tischenko <[email protected]> * Update documentation/add_compiler.md Co-authored-by: Dimitri Tischenko <[email protected]> * Update documentation/add_compiler.md Co-authored-by: Dimitri Tischenko <[email protected]> * Update documentation/add_compiler.md Co-authored-by: Dimitri Tischenko <[email protected]> * Update documentation/add_compiler.md Co-authored-by: Dimitri Tischenko <[email protected]> * Update documentation/add_compiler.md Co-authored-by: Dimitri Tischenko <[email protected]> * Updating add compiler docs to show optional params * Updating docs with text corrections --------- Co-authored-by: Dimitri Tischenko <[email protected]> Co-authored-by: Neil Anderson <[email protected]> * chore(workflows): simplify YAML syntax and improve readability - Removed unnecessary quotes around strings in YAML files. - Consolidated list items to single-line format where applicable. - Ensured consistent naming conventions for job steps. - Improved readability by removing extra blank lines. * fix(workflow): add console_password to test-add-compiler workflows - Added `console_password` parameter to the `test-add-compiler-matrix.yml` and `test-add-compiler.yaml` workflows. - This ensures the console password is passed correctly during the workflow execution. * chore: update test-add-compiler-matrix workflow - Remove centos-7 from the image matrix in test-add-compiler-matrix.yml - Retain only almalinux-cloud/almalinux-8 in the image matrix * feat(workflow): add --stream option to bolt plan run command Added the --stream option to the bolt plan run command in the test-add-compiler GitHub workflow. This change ensures that the output is streamed in real-time. * chore(workflow): update compiler versions in test-add-compiler-matrix.yml Updated the compiler versions in the test-add-compiler-matrix.yml GitHub workflow from 2021.7.7 and 2023.6.0 to 2021.7.9 and 2023.8.0. --------- Co-authored-by: Jo Lillie <[email protected]> Co-authored-by: Neil Anderson <[email protected]> Co-authored-by: Neil Anderson <[email protected]> Co-authored-by: cathal41 <[email protected]> Co-authored-by: Dimitri Tischenko <[email protected]>
1 parent 33b5dac commit 68329f0

File tree

9 files changed

+394
-65
lines changed

9 files changed

+394
-65
lines changed

Diff for: ‎.github/workflows/test-add-compiler-matrix.yml

+105
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
---
2+
name: Add compiler matrix
3+
on:
4+
pull_request:
5+
paths:
6+
- .github/workflows/**/*
7+
- spec/**/*
8+
- lib/**/*
9+
- tasks/**/*
10+
- functions/**/*
11+
- types/**/*
12+
- plans/**/*
13+
- hiera/**/*
14+
- manifests/**/*
15+
- templates/**/*
16+
- files/**/*
17+
- metadata.json
18+
- Rakefile
19+
- Gemfile
20+
- provision.yaml
21+
- .rspec
22+
- .rubocop.yml
23+
- .puppet-lint.rc
24+
- .fixtures.yml
25+
branches: [main]
26+
workflow_dispatch: {}
27+
jobs:
28+
test-add-compiler:
29+
name: PE ${{ matrix.version }} ${{ matrix.architecture }} on ${{ matrix.image }}
30+
runs-on: ubuntu-20.04
31+
env:
32+
BOLT_GEM: true
33+
BOLT_DISABLE_ANALYTICS: true
34+
LANG: en_US.UTF-8
35+
strategy:
36+
fail-fast: false
37+
matrix:
38+
architecture: [standard, large, extra-large]
39+
version: [2021.7.9, 2023.8.0]
40+
image: [almalinux-cloud/almalinux-8]
41+
steps:
42+
- name: Checkout Source
43+
uses: actions/checkout@v2
44+
- name: Activate Ruby 2.7
45+
uses: ruby/setup-ruby@v1
46+
with:
47+
ruby-version: '2.7'
48+
bundler-cache: true
49+
- name: Print bundle environment
50+
if: ${{ github.repository_owner == 'puppetlabs' }}
51+
run: |
52+
echo ::group::info:bundler
53+
bundle env
54+
echo ::endgroup::
55+
- name: Provision test cluster
56+
timeout-minutes: 15
57+
run: |
58+
echo ::group::prepare
59+
mkdir -p $HOME/.ssh
60+
echo 'Host *' > $HOME/.ssh/config
61+
echo ' ServerAliveInterval 150' >> $HOME/.ssh/config
62+
echo ' ServerAliveCountMax 2' >> $HOME/.ssh/config
63+
bundle exec rake spec_prep
64+
echo ::endgroup::
65+
echo ::group::provision
66+
bundle exec bolt plan run peadm_spec::provision_test_cluster \
67+
--modulepath spec/fixtures/modules \
68+
provider=provision_service \
69+
image=${{ matrix.image }} \
70+
architecture=${{ matrix.architecture }}-with-extra-compiler
71+
echo ::endgroup::
72+
echo ::group::info:request
73+
cat request.json || true; echo
74+
echo ::endgroup::
75+
echo ::group::info:inventory
76+
sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true
77+
echo ::endgroup::
78+
- name: Install PE on test cluster
79+
timeout-minutes: 120
80+
run: |
81+
bundle exec bolt plan run peadm_spec::install_test_cluster \
82+
--inventoryfile spec/fixtures/litmus_inventory.yaml \
83+
--modulepath spec/fixtures/modules \
84+
architecture=${{ matrix.architecture }} \
85+
console_password=${{ secrets.CONSOLE_PASSWORD }} \
86+
version=${{ matrix.version }}
87+
- name: Run add_compiler plan
88+
timeout-minutes: 50
89+
run: |
90+
bundle exec bolt plan run peadm_spec::add_compiler -v \
91+
--inventoryfile spec/fixtures/litmus_inventory.yaml \
92+
--modulepath spec/fixtures/modules \
93+
architecture=${{ matrix.architecture }}
94+
- name: Tear down test cluster
95+
if: ${{ always() }}
96+
continue-on-error: true
97+
run: |-
98+
if [ -f spec/fixtures/litmus_inventory.yaml ]; then
99+
echo ::group::tear_down
100+
bundle exec rake 'litmus:tear_down'
101+
echo ::endgroup::
102+
echo ::group::info:request
103+
cat request.json || true; echo
104+
echo ::endgroup::
105+
fi

Diff for: ‎.github/workflows/test-add-compiler.yaml

+5-4
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ jobs:
5353
echo ::group::info:bundler
5454
bundle env
5555
echo ::endgroup::
56-
- name: Provision test cluster (specified architecture with added DR)
56+
- name: Provision test cluster (specified architecture with added compiler)
5757
timeout-minutes: 15
5858
run: |
5959
echo ::group::prepare
@@ -84,14 +84,15 @@ jobs:
8484
--modulepath spec/fixtures/modules \
8585
architecture=${{ matrix.architecture }} \
8686
version=${{ matrix.version }} \
87-
console_password=${{ secrets.CONSOLE_PASSWORD }}
87+
console_password=${{ secrets.CONSOLE_PASSWORD }
8888
- name: Run add_compiler plan
89-
timeout-minutes: 30
89+
timeout-minutes: 50
9090
run: |
9191
bundle exec bolt plan run peadm_spec::add_compiler -v \
9292
--inventoryfile spec/fixtures/litmus_inventory.yaml \
9393
--modulepath spec/fixtures/modules \
94-
architecture=${{ matrix.architecture }}
94+
architecture=${{ matrix.architecture }} \
95+
--stream
9596
- name: Wait as long as the file ${HOME}/pause file is present
9697
if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }}
9798
run: |

Diff for: ‎REFERENCE.md

+51-1
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@
8888

8989
#### Public Plans
9090

91+
* [`peadm::add_compiler`](#peadm--add_compiler): Add a new compiler to a PE architecture or replace an existing one with new configuration.
9192
* [`peadm::add_database`](#peadm--add_database)
9293
* [`peadm::backup`](#peadm--backup): Backup puppet primary configuration
9394
* [`peadm::backup_ca`](#peadm--backup_ca)
@@ -102,7 +103,6 @@
102103

103104
#### Private Plans
104105

105-
* `peadm::add_compiler`: Add a new compiler to a PE architecture or replace an existing one with new configuration.
106106
* `peadm::add_replica`: Replace a replica host for a Standard or Large architecture.
107107
Supported use cases:
108108
1: The existing replica is broken, we have a fresh new VM we want to provision the replica to.
@@ -1555,6 +1555,56 @@ Which port to query the status API on
15551555

15561556
## Plans
15571557

1558+
### <a name="peadm--add_compiler"></a>`peadm::add_compiler`
1559+
1560+
Add a new compiler to a PE architecture or replace an existing one with new configuration.
1561+
1562+
#### Parameters
1563+
1564+
The following parameters are available in the `peadm::add_compiler` plan:
1565+
1566+
* [`avail_group_letter`](#-peadm--add_compiler--avail_group_letter)
1567+
* [`compiler_host`](#-peadm--add_compiler--compiler_host)
1568+
* [`dns_alt_names`](#-peadm--add_compiler--dns_alt_names)
1569+
* [`primary_host`](#-peadm--add_compiler--primary_host)
1570+
* [`primary_postgresql_host`](#-peadm--add_compiler--primary_postgresql_host)
1571+
1572+
##### <a name="-peadm--add_compiler--avail_group_letter"></a>`avail_group_letter`
1573+
1574+
Data type: `Enum['A', 'B']`
1575+
1576+
_ Either A or B; whichever of the two letter designations the compiler is being assigned to
1577+
1578+
Default value: `'A'`
1579+
1580+
##### <a name="-peadm--add_compiler--compiler_host"></a>`compiler_host`
1581+
1582+
Data type: `Peadm::SingleTargetSpec`
1583+
1584+
_ The hostname and certname of the new compiler
1585+
1586+
##### <a name="-peadm--add_compiler--dns_alt_names"></a>`dns_alt_names`
1587+
1588+
Data type: `Optional[String[1]]`
1589+
1590+
_ A comma_separated list of DNS alt names for the compiler
1591+
1592+
Default value: `undef`
1593+
1594+
##### <a name="-peadm--add_compiler--primary_host"></a>`primary_host`
1595+
1596+
Data type: `Peadm::SingleTargetSpec`
1597+
1598+
_ The hostname and certname of the primary Puppet server
1599+
1600+
##### <a name="-peadm--add_compiler--primary_postgresql_host"></a>`primary_postgresql_host`
1601+
1602+
Data type: `Optional[Peadm::SingleTargetSpec]`
1603+
1604+
_ The hostname and certname of the PE-PostgreSQL server with availability group $avail_group_letter
1605+
1606+
Default value: `undef`
1607+
15581608
### <a name="peadm--add_database"></a>`peadm::add_database`
15591609

15601610
The peadm::add_database class.

Diff for: ‎documentation/add_compiler.md

+64
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Add Compiler
2+
3+
- [Add Compiler](#Add-Compiler)
4+
- [Introduction](#Introduction)
5+
- [Add a compiler to an existing PE cluster](#Add-a-compiler-to-an-existing-PE-cluster)
6+
- [Running the `add_compiler` plan](#running-the-add_compiler-plan)
7+
- [Optional Parameters](#optional-parameters)
8+
9+
10+
## Introduction
11+
12+
The `peadm::add_compiler` plan can be used to add a new compiler to a PE cluster or replace an existing one with new configuration.
13+
14+
15+
## Add a compiler to an existing PE cluster
16+
17+
As seen in the example below, this is the minimal parameters required to add a compiler to an existing PE cluster. These can be passed as command line variables to the plan, or in this case added to a params.json file.
18+
19+
```json
20+
{
21+
"compiler_host": "pe-xl-compiler-0.lab1.puppet.vm",
22+
"primary_host": "pe-xl-core-0.lab1.puppet.vm"
23+
}
24+
```
25+
26+
## Running the `add_compiler` plan
27+
```
28+
bolt plan run peadm::add_compiler --params @params.json
29+
```
30+
31+
This command will retrieve the current PEADM configuration to determine the setup rules needed for a compiler's secondary PuppetDB instances. The plan will configure the primary with appropriate rules for allowing access from the new compiler. On the primary, the `puppet` service is stopped and the `pe-postgresql` service is reloaded. If required, a puppet agent will be installed on the new compiler host. The compiler agent's certificate is regenerated to include data required by the `peadm::subplans::component_install` plan. A subsequent Puppet agent run will happen on the following components.
32+
* _\<compiler-host\>_
33+
* _\<primary_postgresql_host\>_
34+
* _\<replica host\>_
35+
* _\<primary_postgresql_host\>_
36+
37+
The `puppet` service is then restarted.
38+
39+
40+
## Optional Parameters
41+
42+
As well as `compiler_host` and `primary_host`, the `add_compiler` plan has a number of optional parameters. These can be viewed in the following params example.
43+
44+
```json
45+
{
46+
"avail_group_letter": "B",
47+
"compiler_host": "pe-xl-compiler-0.lab1.puppet.vm",
48+
"dns_alt_names": [ "puppet", "puppet.lab1.puppet.vm" ],
49+
"primary_host": "pe-xl-core-0.lab1.puppet.vm",
50+
"primary_postgresql_host": "pe-psql-6251cd-0.us-west1-a.c.slice-cody.internal"
51+
}
52+
```
53+
54+
Please note, the optional parameters and values of the plan are as follows.
55+
56+
<!-- table -->
57+
| Parameter | Default value | Description |
58+
|--------------------------|---------------|--------------------------------------------------|
59+
| `avail_group_letter` | `A` | By default, each compiler will be added to the primary group A. |
60+
| `dns_alt_names` | `undef` | |
61+
| `primary_postgresql_host`| `undef` | By default, this will pre-populate to the required value depending if your architecture contains HA and or external databases. |
62+
63+
For more information around adding compilers to your infrastructure [Expanding Your Deployment](expanding.md#adding-compilers-with-peadmadd_compiler)
64+

Diff for: ‎documentation/expanding.md

+7-7
Original file line numberDiff line numberDiff line change
@@ -101,21 +101,21 @@ The Standard deployment architecture is the only deployment architecture of the
101101

102102
### Adding Compilers to Standard and Large without disaster recovery
103103

104-
The command invocation is identical when adding Compilers to a Standard or Large deployment architecture if disaster recovery (DR) is not enabled and a Replica Puppet server has not been provisioned. Take note of the values for `avail_group_letter` and `primary_postgresql_host`, in this **no DR** scenario, the value of these parameters will always be set to **A** and the FQDN of the Primary Puppet server.
104+
The command invocation is identical when adding Compilers to a Standard or Large deployment architecture if disaster recovery (DR) is not enabled and a replica Puppet server has not been provisioned. Take note that `avail_group_letter` is not required in this **no DR** scenario. By default, the value of this parameter is set to **A**.
105105

106-
bolt plan run peadm::add_compiler primary_host=<primary-server-fqdn> compiler_host=<new-compiler-fqdn> avail_group_letter=A primary_postgresql_host=<primary-server-fqdn>
106+
bolt plan run peadm::add_compiler primary_host=<primary-server-fqdn> compiler_host=<new-compiler-fqdn>
107107

108108
### Adding Compilers to Extra Large without disaster recovery
109109

110-
When adding a Compiler to a deployment which has adopted the Extra Large deployment architecture in a **no DR** scenario, the only difference is that the `primary_postgresql_host` changes to be the value of the Primary PE-PostgreSQL server as opposed to the Primary Puppet server.
110+
When adding a compiler to a deployment which has adopted the Extra Large deployment architecture in a **no DR** scenario, the only difference is that the `primary_postgresql_host` changes to the value of the primary PE-PostgreSQL server as opposed to the Primary Puppet server.
111111

112-
bolt plan run peadm::add_compiler primary_host=<primary-server-fqdn> compiler_host=<new-compiler-fqdn> avail_group_letter=A primary_postgresql_host=<primary-postgresql-server-fqdn>
112+
bolt plan run peadm::add_compiler primary_host=<primary-server-fqdn> compiler_host=<new-compiler-fqdn>
113113

114114
### Adding Compilers to Standard and Large when disaster recovery has been enabled
115115

116-
As was described in the section documenting [peadm::add_replica](#enable-disaster-recovery-and-add-a-replica-with-peadmadd_replica), when disaster recovery (DR) is enabled and a Replica provisioned, PEADM creates a second availability group, **B**. You must take this second availability group into consideration when adding new compilers and ensure you are assigning appropriate values for the group the Compiler is targeted for. It is a good idea to keep these two availability groups populated with an equal quantity of Compilers. Besides the value of `avail_group_letter` being dependent on which group the new Compiler is targeted towards, the value of `primary_postgresql_host` will also vary.
116+
As was described in the section documenting [peadm::add_replica](#enable-disaster-recovery-and-add-a-replica-with-peadmadd_replica), when disaster recovery (DR) is enabled and a Replica provisioned, PEADM creates a second availability group, **B**. You must take this second availability group into consideration when adding new compilers and ensure you are assigning appropriate values for the group the compiler is targeted for. It is a good idea to keep these two availability groups populated with an equal quantity of compilers. Besides the value of `avail_group_letter` being dependent on which group the new compiler is targeted towards, the value of `primary_postgresql_host` will also vary.
117117

118-
The name of the `primary_postgresql_host` parameter can be confusing, it is **NOT** always equal to the Primary Puppet server or Primary PE-PostgreSQL server, it can also be equal to the Replica Puppet server or Replica PE-PostgreSQL server. It should be set to the server which is a member of the Compiler's target availability group. The easiest way to determine this value is to first run the `peadm::get_peadm_config` task and source the value from its output. In the **Example** section the value to use when targeting the **B** group is `pe-server-59ab63-1.us-west1-b.c.slice-cody.internal`. You'll find the value at `role-letter.server.B`.
118+
The name of the `primary_postgresql_host` parameter can be confusing, it is **NOT** always equal to the Primary Puppet server or Primary PE-PostgreSQL server, it can also be equal to the replica Puppet server or replica PE-PostgreSQL server. It should be set to the server which is a member of the compiler's target availability group. In most cases this will be handled behind the scenes and not be required to be worked out by the user. The easiest way to determine this value is to first run the `peadm::get_peadm_config` task and source the value from its output. In the **Example** section the value to use when targeting the **B** group is `pe-server-59ab63-1.us-west1-b.c.slice-cody.internal`. You'll find the value at `role-letter.server.B`.
119119

120120
bolt plan run peadm::get_peadm_config --targets <primary-server-fqdn>
121121
bolt plan run peadm::add_compiler primary_host=<primary-server-fqdn> compiler_host=<new-compiler-fqdn> avail_group_letter=<new-compiler-target-group> primary_postgresql_host=<target-group-server-fqdn>
@@ -161,7 +161,7 @@ The name of the `primary_postgresql_host` parameter can be confusing, it is **NO
161161
Successful on 1 target: pe-server-59ab63-0.us-west1-a.c.slice-cody.internal
162162
Ran on 1 target in 2.46 sec
163163

164-
### Adding compilers to Extra Large when disaster recovery has been enabled
164+
### Adding Compilers to Extra Large when disaster recovery has been enabled
165165

166166
Adding a Compiler to a deployment which has adopted the Extra Large deployment architecture with disaster recovery (DR) enabled is similar to Standard and Large but the value of `primary_postgresql_host` will no longer correspond to the Primary or Replica Puppet server since PuppetDB databases are now hosted externally. In the **Example** section, the value to use when targeting the **A** group is `pe-psql-65e03f-0.us-west1-a.c.slice-cody.internal`. You'll find the value at `role-letter.postgresql.A`.
167167

Diff for: ‎plans/add_compiler.pp

+29-10
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
1-
# This plan is in development and currently considered experimental.
2-
#
3-
# @api private
1+
# @api public
42
#
53
# @summary Add a new compiler to a PE architecture or replace an existing one with new configuration.
64
# @param avail_group_letter _ Either A or B; whichever of the two letter designations the compiler is being assigned to
@@ -9,20 +7,41 @@
97
# @param primary_host _ The hostname and certname of the primary Puppet server
108
# @param primary_postgresql_host _ The hostname and certname of the PE-PostgreSQL server with availability group $avail_group_letter
119
plan peadm::add_compiler(
12-
Enum['A', 'B'] $avail_group_letter,
10+
Enum['A', 'B'] $avail_group_letter = 'A' ,
1311
Optional[String[1]] $dns_alt_names = undef,
1412
Peadm::SingleTargetSpec $compiler_host,
1513
Peadm::SingleTargetSpec $primary_host,
16-
Peadm::SingleTargetSpec $primary_postgresql_host,
14+
Optional[Peadm::SingleTargetSpec] $primary_postgresql_host = undef,
1715
) {
1816
$compiler_target = peadm::get_targets($compiler_host, 1)
1917
$primary_target = peadm::get_targets($primary_host, 1)
20-
$primary_postgresql_target = peadm::get_targets($primary_postgresql_host, 1)
2118

2219
# Get current peadm config to determine where to setup additional rules for
2320
# compiler's secondary PuppetDB instances
2421
$peadm_config = run_task('peadm::get_peadm_config', $primary_target).first.value
2522

23+
if $primary_postgresql_host == undef {
24+
# get the external PostgreSQL host for the specified availability group
25+
$external_postgresql_host = $avail_group_letter ? {
26+
'A' => $peadm_config['params']['primary_postgresql_host'],
27+
default => $peadm_config['params']['replica_postgresql_host'],
28+
}
29+
30+
# If the external_postgresql_host is undef, use the server for that availability group
31+
$postgresql_host = $external_postgresql_host ? {
32+
undef => $peadm_config['role-letter']['server'][$avail_group_letter],
33+
default => $external_postgresql_host,
34+
}
35+
36+
if $postgresql_host == undef {
37+
fail_plan("No PostgreSQL host found for availability group ${avail_group_letter}")
38+
}
39+
40+
$primary_postgresql_target = peadm::get_targets($postgresql_host, 1)
41+
} else {
42+
$primary_postgresql_target = peadm::get_targets($primary_postgresql_host, 1)
43+
}
44+
2645
# Return the opposite server than the compiler to be added so it can be
2746
# configured with the appropriate rules for Puppet Server access from
2847
# compiler
@@ -84,16 +103,16 @@
84103
path => '/etc/puppetlabs/puppet/hiera.yaml'
85104
)
86105

87-
# On <compiler-host>, run the puppet agent
106+
# On <compiler-host>, run puppet agent
88107
run_task('peadm::puppet_runonce', $compiler_target)
89108

90-
# On <primary_postgresql_host> run the puppet agent
109+
# On <primary_postgresql_host>, run puppet agent
91110
run_task('peadm::puppet_runonce', $primary_postgresql_target)
92111

93-
# On replica puppetdb run the puppet agent
112+
# On <replica_puppetdb_target>, run puppet agent
94113
run_task('peadm::puppet_runonce', $replica_puppetdb_target)
95114

96-
# On <primary_postgresql_host> start puppet.service
115+
# On <primary_postgresql_host>, start puppet.service
97116
run_command('systemctl start puppet.service', peadm::flatten_compact([
98117
$primary_postgresql_target,
99118
$replica_puppetdb_target,

0 commit comments

Comments
 (0)