From 564cafe7f6f9735ea6b8dfdb2b1f6a0e90070bb9 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 17 Feb 2023 16:35:10 +0100 Subject: [PATCH 001/110] Re-add backup/restore plans --- documentation/backup.md | 74 ++++++++++++ documentation/restore.md | 46 +++++++ plans/backup.pp | 123 +++++++++++++++++++ plans/restore.pp | 238 +++++++++++++++++++++++++++++++++++++ spec/plans/backup_spec.rb | 22 ++++ spec/plans/restore_spec.rb | 24 ++++ 6 files changed, 527 insertions(+) create mode 100644 documentation/backup.md create mode 100644 documentation/restore.md create mode 100644 plans/backup.pp create mode 100644 plans/restore.pp create mode 100644 spec/plans/backup_spec.rb create mode 100644 spec/plans/restore_spec.rb diff --git a/documentation/backup.md b/documentation/backup.md new file mode 100644 index 00000000..5e19daa3 --- /dev/null +++ b/documentation/backup.md @@ -0,0 +1,74 @@ +# Backup Puppet Enterprise using the PEADM module + +## What is being backed up? + +By default the `peadm::backup` plan will backup the following items: + +1. Orchestrator +2. PuppetDB +3. RBAC + - Also, It copies the LDAP secret key if it present +4. Activity +5. Classification + +Optionally you can also backup: + +1. CA (CA and SSL certificates) + +---- + +Most of the backups will be a direct copy of the databases with the exception of: + +- Classification + - The backup is done using an API call +- CA + - The certificate files will be copy via the puppet-backup script. + + +**Note:** + +It is important to highlight that the `peadm::backup` plan's output is different than the one you will get when you backup manually using [the `puppet-backup create` command.](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure). + +The differences between these two backup tools are: + +1. The structure of the backup file, since this plan `peadm:backup` uses a combination of scripts, API calls, and DB backups, you will not be able to restore it using the traditional `sudo puppet-backup restore ` command. [To read more about the difference between the options (flags), please read the official PE doc for backing up & restoring.](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure) + +1. The core idea of the `peadm:backup` plan is a focus on being able to separate customers data from the infrastructure data. This contrasts with the PE backup script which is more focused on a traditional backup and restore after a system failure where will be restoring the same infrastructure. It, therefore, targets things like backing up PostgreSQL rather than the actual data you want. + - This can be seen when restoring node groups with `peadm:backup` only restores the non-PEinfrastructure source groups and restores them to target. + + - [PE backup](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure) is not capable of restoring to a DR setup and requires you to provision a new DR setup whereas [`peadm:restore`](restore.md) will reinitialise replicas as part of the process + +1. One of the other key differences with the `peadm:backup` is it is compatible with the extra-large (XL) architectures as it can communicate with the remote PostgreSQL server using temporary escalated privileges. + +## How can I customize my backup? + +We need to pass the `backup` parameter to the `peadm::backup` plan. + +**Example** + +**Note:** The `peadm::backup` plan can only be executed from the PE primary server. + +Let's Backup _only_ RBAC + +``` +# backup_params.json + +{ + "backup": { + "orchestrator": false, + "puppetdb": false, + "rbac": true, + "activity": false, + "ca": false, + "classifier": false + }, + "output_directory": "/tmp" +} +``` + +We selected our backup options and the `output_directory` (default `/tmp`). + +To run the backup plan with our custom parameters: + + bolt plan run peadm::backup --params @backup_params.json + diff --git a/documentation/restore.md b/documentation/restore.md new file mode 100644 index 00000000..50331ead --- /dev/null +++ b/documentation/restore.md @@ -0,0 +1,46 @@ +# Restore Puppet Enterprise using the PEADM module + +Once you have a [backup](backup.md) you can restore a PE primary server. It is important to highlight that you can not use the `peadm::restore` plan with a backup that was not created with the `peadm::backup` plan. + +**Things to consider** + +There is some downtime involved when the `peadm:restore` plan is executed. The following services will be restarted: + +1. pe-console-services +2. pe-nginx +3. pxp-agent +4. pe-puppetserver +5. pe-orchestration-services +6. puppet +7. pe-puppetdb + +There is also a procedure related to the restoration of the databases where `peadm::restore` will temporarily set privileges (permissions) to a DB user with the only purpose of restoring the database. These temporary privileges are removed right after the PostgreSQL restore command finishes. + +Also, this plan uses internal calls to the `peadm::get_targets` function, this means the plan expects the services to be up and running when you start either a restore or a backup. + +## How to use the restore plan? + +As in the `peadm::backup` plan, you can choose what you want to restore by specifying the parameter `restore`. The `input_file` parameter refers to the location of the backup tarball. + +Example: + +**Note:** The `peadm::restore` plan can only be executed from the PE primary server. + +``` +# restore_params.json + +{ + "restore": { + "orchestrator": false, + "puppetdb": false, + "rbac": true, + "activity": false, + "ca": false, + "classifier": false, + }, + "input_file": "/tmp/path/backup_tarball.tgz" +} +``` +To run the `peadm::restore` plan with our custom parameters file, we can do: + + bolt plan run peadm::restore --params @restore_params.json diff --git a/plans/backup.pp b/plans/backup.pp new file mode 100644 index 00000000..b653e99f --- /dev/null +++ b/plans/backup.pp @@ -0,0 +1,123 @@ +# @api private +# @summary Backup the core user settings for puppet infrastructure +# +# This plan can backup data as outlined at insert doc +# +plan peadm::backup ( + # This plan should be run on the primary server + Peadm::SingleTargetSpec $targets, + + # Which data to backup + Peadm::Recovery_opts $backup = {}, + + # Where to put the backup folder + String $output_directory = '/tmp', +) { + peadm::assert_supported_bolt_version() + + $recovery_opts = (peadm::recovery_opts_default() + $backup) + $cluster = run_task('peadm::get_peadm_config', $targets).first.value + $arch = peadm::assert_supported_architecture( + getvar('cluster.params.primary_host'), + getvar('cluster.params.replica_host'), + getvar('cluster.params.primary_postgresql_host'), + getvar('cluster.params.replica_postgresql_host'), + getvar('cluster.params.compiler_hosts'), + ) + + $timestamp = Timestamp.new().strftime('%Y-%m-%dT%H%M%SZ') + $backup_directory = "${output_directory}/pe-backup-${timestamp}" + + $primary_target = getvar('cluster.params.primary_host') + $puppetdb_postgresql_target = getvar('cluster.params.primary_postgresql_host') ? { + undef => getvar('cluster.params.primary_host'), + default => getvar('cluster.params.primary_postgresql_host'), + } + + $backup_databases = { + 'orchestrator' => $primary_target, + 'activity' => $primary_target, + 'rbac' => $primary_target, + 'puppetdb' => $puppetdb_postgresql_target, + }.filter |$key,$_| { + $recovery_opts[$key] == true + } + + # Create backup folders + apply($primary_target) { + file { $backup_directory : + ensure => 'directory', + owner => 'root', + group => 'root', + mode => '0700', + } + + # Create a subdir for each backup type selected + $recovery_opts.filter |$_,$val| { $val == true }.each |$dir,$_| { + file { "${backup_directory}/${dir}": + ensure => 'directory', + owner => 'root', + group => 'root', + mode => '0700', + } + } + } + + if getvar('recovery_opts.classifier') { + out::message('# Backing up classification') + run_task('peadm::backup_classification', $primary_target, + directory => "${backup_directory}/classifier", + ) + } + + if getvar('recovery_opts.ca') { + out::message('# Backing up ca and ssl certificates') +# lint:ignore:strict_indent + run_command(@("CMD"), $primary_target) + /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/ca --scope=certs + | CMD + } + + # Check if /etc/puppetlabs/console-services/conf.d/secrets/keys.json exists and if so back it up + if getvar('recovery_opts.rbac') { + out::message('# Backing up ldap secret key if it exists') +# lint:ignore:140chars + run_command(@("CMD"/L), $primary_target) + test -f /etc/puppetlabs/console-services/conf.d/secrets/keys.json \ + && cp -rp /etc/puppetlabs/console-services/conf.d/secrets ${shellquote($backup_directory)}/rbac/ \ + || echo secret ldap key doesnt exist + | CMD +# lint:endignore + } +# lint:ignore:140chars + # IF backing up orchestrator back up the secrets too /etc/puppetlabs/orchestration-services/conf.d/secrets/ + if getvar('recovery_opts.orchestrator') { + out::message('# Backing up orchestrator secret keys') + run_command(@("CMD"), $primary_target) + cp -rp /etc/puppetlabs/orchestration-services/conf.d/secrets ${shellquote($backup_directory)}/orchestrator/ + | CMD + } +# lint:endignore + $backup_databases.each |$name,$database_target| { + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 \ + -f ${shellquote($backup_directory)}/${shellquote($name)}/pe-${shellquote($name)}.dump.d \ + "sslmode=verify-ca \ + host=${shellquote($database_target.peadm::certname())} \ + user=pe-${shellquote($name)} \ + sslcert=/etc/puppetlabs/puppetdb/ssl/${shellquote($primary_target.peadm::certname())}.cert.pem \ + sslkey=/etc/puppetlabs/puppetdb/ssl/${shellquote($primary_target.peadm::certname())}.private_key.pem \ + sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem \ + dbname=pe-${shellquote($name)}" + | CMD + } + + run_command(@("CMD"/L), $primary_target) + umask 0077 \ + && cd ${shellquote(dirname($backup_directory))} \ + && tar -czf ${shellquote($backup_directory)}.tar.gz ${shellquote(basename($backup_directory))} \ + && rm -rf ${shellquote($backup_directory)} + | CMD +# lint:endignore + return({ 'path' => "${backup_directory}.tar.gz" }) +} diff --git a/plans/restore.pp b/plans/restore.pp new file mode 100644 index 00000000..2bbe7d8c --- /dev/null +++ b/plans/restore.pp @@ -0,0 +1,238 @@ +# @api private +# @summary Restore the core user settings for puppet infrastructure from backup +# +# This plan can restore data to puppet infrastructure for DR and rebuilds +# +plan peadm::restore ( + # This plan should be run on the primary server + Peadm::SingleTargetSpec $targets, + + # Which data to restore + Peadm::Recovery_opts $restore = {}, + + # Path to the recovery tarball + Pattern[/.*\.tar\.gz$/] $input_file, +) { + peadm::assert_supported_bolt_version() + + $recovery_opts = (peadm::recovery_opts_default() + $restore) + $cluster = run_task('peadm::get_peadm_config', $targets).first.value + $arch = peadm::assert_supported_architecture( + getvar('cluster.params.primary_host'), + getvar('cluster.params.replica_host'), + getvar('cluster.params.primary_postgresql_host'), + getvar('cluster.params.replica_postgresql_host'), + getvar('cluster.params.compiler_hosts'), + ) + + $primary_target = peadm::get_targets(getvar('cluster.params.primary_host'), 1) + $replica_target = peadm::get_targets(getvar('cluster.params.replica_host'), 1) + $compiler_targets = peadm::get_targets(getvar('cluster.params.compiler_hosts')) + + # Determine the array of targets to which the PuppetDB PostgreSQL database + # should be restored to. This could be as simple as just the primary server, + # or it could be two separate PostgreSQL servers. + $puppetdb_postgresql_targets = peadm::flatten_compact([ + getvar('cluster.params.primary_postgresql_host') ? { + undef => $primary_target, + default => peadm::get_targets(getvar('cluster.params.primary_postgresql_host'), 1), + }, + getvar('cluster.params.replica_postgresql_host') ? { + undef => $replica_target, + default => peadm::get_targets(getvar('cluster.params.replica_postgresql_host'), 1), + }, + ]) + + $puppetdb_targets = peadm::flatten_compact([ + $primary_target, + $replica_target, + $compiler_targets, + ]) + + $recovery_directory = "${dirname($input_file)}/${basename($input_file, '.tar.gz')}" +# lint:ignore:strict_indent + run_command(@("CMD"/L), $primary_target) + umask 0077 \ + && cd ${shellquote(dirname($recovery_directory))} \ + && tar -xzf ${shellquote($input_file)} + | CMD +# lint:endignore + # Map of recovery option name to array of database hosts to restore the + # relevant .dump content to. + $restore_databases = { + 'orchestrator' => [$primary_target], + 'activity' => [$primary_target], + 'rbac' => [$primary_target], + 'puppetdb' => $puppetdb_postgresql_targets, + }.filter |$key,$_| { + $recovery_opts[$key] == true + } + + if getvar('recovery_opts.classifier') { + out::message('# Restoring classification') + run_task('peadm::backup_classification', $primary_target, + directory => $recovery_directory + ) + out::message("# Backed up current classification to ${recovery_directory}/classification_backup.json") + + run_task('peadm::transform_classification_groups', $primary_target, + source_directory => "${recovery_directory}/classifier", + working_directory => $recovery_directory + ) + + run_task('peadm::restore_classification', $primary_target, + classification_file => "${recovery_directory}/classification_backup.json", + ) + } + + if getvar('recovery_opts.ca') { + out::message('# Restoring ca and ssl certificates') +# lint:ignore:strict_indent + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/bin/puppet-backup restore \ + --scope=certs \ + --tempdir=${shellquote($recovery_directory)} \ + --force \ + ${shellquote($recovery_directory)}/classifier/pe_backup-*tgz + | CMD + } + + # Use PuppetDB's /pdb/admin/v1/archive API to SAVE data currently in PuppetDB. + # Otherwise we'll completely lose it if/when we restore. + # TODO: consider adding a heuristic to skip when innappropriate due to size + # or other factors. + if getvar('recovery_opts.puppetdb') { + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/bin/puppet-db export ${shellquote($recovery_directory)}/puppetdb-archive.bin + | CMD + } + + ## shutdown services + run_command(@("CMD"/L), $primary_target) + systemctl stop pe-console-services pe-nginx pxp-agent pe-puppetserver \ + pe-orchestration-services puppet pe-puppetdb + | CMD + + # Restore secrets/keys.json if it exists + out::message('# Restoring ldap secret key if it exists') + run_command(@("CMD"/L), $primary_target) + test -f ${shellquote($recovery_directory)}/rbac/keys.json \ + && cp -rp ${shellquote($recovery_directory)}/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ \ + || echo secret ldap key doesnt exist + | CMD +# lint:ignore:140chars + # IF restoring orchestrator restore the secrets to /etc/puppetlabs/orchestration-services/conf.d/secrets/ + if getvar('recovery_opts.orchestrator') { + out::message('# Restoring orchestrator secret keys') + run_command(@("CMD"/L), $primary_target) + cp -rp ${shellquote($recovery_directory)}/orchestrator/secrets/* /etc/puppetlabs/orchestration-services/conf.d/secrets/ + | CMD + } +# lint:endignore + + #$database_to_restore.each |Integer $index, Boolean $value | { + $restore_databases.each |$name,$database_targets| { + out::message("# Restoring ${name} database") + $dbname = "pe-${shellquote($name)}" + + # Drop pglogical extensions and schema if present + run_command(@("CMD"/L), $database_targets) + su - pe-postgres -s /bin/bash -c \ + "/opt/puppetlabs/server/bin/psql \ + --tuples-only \ + -d '${dbname}' \ + -c 'DROP SCHEMA IF EXISTS pglogical CASCADE;'" + | CMD + + run_command(@("CMD"/L), $database_targets) + su - pe-postgres -s /bin/bash -c \ + "/opt/puppetlabs/server/bin/psql \ + -d '${dbname}' \ + -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'" + | CMD + + # To allow db user to restore the database grant temporary privileges + run_command(@("CMD"/L), $database_targets) + su - pe-postgres -s /bin/bash -c \ + "/opt/puppetlabs/server/bin/psql \ + -d '${dbname}' \ + -c 'ALTER USER \"${dbname}\" WITH SUPERUSER;'" + | CMD + + # Restore database. If there are multiple database restore targets, perform + # the restore(s) in parallel. + parallelize($database_targets) |$database_target| { + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/server/bin/pg_restore \ + -j 4 \ + -d "sslmode=verify-ca \ + host=${shellquote($database_target.peadm::certname())} \ + sslcert=/etc/puppetlabs/puppetdb/ssl/${shellquote($primary_target.peadm::certname())}.cert.pem \ + sslkey=/etc/puppetlabs/puppetdb/ssl/${shellquote($primary_target.peadm::certname())}.private_key.pem \ + sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem \ + dbname=${dbname} \ + user=${dbname}" \ + -Fd ${recovery_directory}/${name}/${dbname}.dump.d + | CMD + } + + # Remove db user privileges post restore + run_command(@("CMD"/L), $database_targets) + su - pe-postgres -s /bin/bash -c \ + "/opt/puppetlabs/server/bin/psql \ + -d '${dbname}' \ + -c 'ALTER USER \"${dbname}\" WITH NOSUPERUSER;'" + | CMD + + # Drop pglogical extension and schema (again) if present after db restore + run_command(@("CMD"/L), $database_targets) + su - pe-postgres -s /bin/bash -c \ + "/opt/puppetlabs/server/bin/psql \ + --tuples-only \ + -d '${dbname}' \ + -c 'DROP SCHEMA IF EXISTS pglogical CASCADE;'" + | CMD + + run_command(@("CMD"/L), $database_targets) + su - pe-postgres -s /bin/bash -c \ + "/opt/puppetlabs/server/bin/psql \ + -d '${dbname}' \ + -c 'DROP EXTENSION IF EXISTS pglogical CASCADE;'" + | CMD + } + + # Use `puppet infra` to ensure correct file permissions, restart services, + # etc. Make sure not to try and get config data from the classifier, which + # isn't yet up and running. + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/bin/puppet-infrastructure configure --no-recover + | CMD + + # If we have replicas reinitalise them + run_command(@("CMD"/L), $replica_target) + /opt/puppetlabs/bin/puppet-infra reinitialize replica -y + | CMD + + # Use PuppetDB's /pdb/admin/v1/archive API to MERGE previously saved data + # into the restored database. + # TODO: consider adding a heuristic to skip when innappropriate due to size + # or other factors. + if getvar('recovery_opts.puppetdb') { + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/bin/puppet-db import ${shellquote($recovery_directory)}/puppetdb-archive.bin + | CMD +# lint:endignore + } + + # Run Puppet to pick up last remaining config tweaks + run_task('peadm::puppet_runonce', $primary_target) + + apply($primary_target) { + file { $recovery_directory : + ensure => 'absent', + force => true, + } + } + + return('success') +} diff --git a/spec/plans/backup_spec.rb b/spec/plans/backup_spec.rb new file mode 100644 index 00000000..f16caecc --- /dev/null +++ b/spec/plans/backup_spec.rb @@ -0,0 +1,22 @@ +require 'spec_helper' + +describe 'peadm::backup' do + include BoltSpec::Plans + let(:params) { { 'targets' => 'primary' } } + + it 'runs with default params' do + allow_apply + pending('a lack of support for functions requires a workaround to be written') + expect_task('peadm::get_peadm_config').always_return({ 'primary_postgresql_host' => 'postgres' }) + expect_out_message.with_params('# Backing up ca and ssl certificates') + # The commands all have a timestamp in them and frankly its proved to hard with bolt spec to work this out + allow_any_command + allow_apply + expect_out_message.with_params('# Backing up database pe-orchestrator') + expect_out_message.with_params('# Backing up database pe-activity') + expect_out_message.with_params('# Backing up database pe-rbac') + expect_out_message.with_params('# Backing up classification') + expect_task('peadm::backup_classification') + expect(run_plan('peadm::backup', params)).to be_ok + end +end diff --git a/spec/plans/restore_spec.rb b/spec/plans/restore_spec.rb new file mode 100644 index 00000000..adceca3d --- /dev/null +++ b/spec/plans/restore_spec.rb @@ -0,0 +1,24 @@ +require 'spec_helper' + +describe 'peadm::restore' do + include BoltSpec::Plans + let(:params) { { 'targets' => 'primary', 'backup_timestamp' => '2022-03-29_16:57:41' } } + + it 'runs with default params' do + allow_apply + pending('a lack of support for functions requires a workaround to be written') + expect_task('peadm::get_peadm_config').always_return({ 'primary_postgresql_host' => 'postgres' }) + expect_out_message.with_params('# Backing up ca and ssl certificates') + # The commands all have a timestamp in them and frankly its proved to hard with bolt spec to work this out + allow_any_command + expect_out_message.with_params('# Restoring classification') + expect_out_message.with_params('# Backed up current classification to /tmp/classification_backup.json') + expect_out_message.with_params('# Restoring ca and ssl certificates') + expect_out_message.with_params('# Restoring database pe-orchestrator') + expect_out_message.with_params('# Restoring database pe-activity') + expect_out_message.with_params('# Restoring database pe-rbac') + expect_out_message.with_params('# Restoring classification') + expect_task('peadm::backup_classification') + expect(run_plan('peadm::restore', params)).to be_ok + end +end From 9e2958968056d2d9782ab794a7fd881d1ec15376 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Wed, 12 Apr 2023 15:09:20 +0200 Subject: [PATCH 002/110] first WIP version of test plan and workflow --- .github/workflows/test-backup-restore.yaml | 206 ++++++++++++------ .../peadm_spec/plans/test_backup.pp | 24 ++ 2 files changed, 162 insertions(+), 68 deletions(-) create mode 100644 spec/acceptance/peadm_spec/plans/test_backup.pp diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 16f9ef82..922a00c6 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -21,28 +21,85 @@ on: required: true default: 'false' -env: - HONEYCOMB_WRITEKEY: 7f3c63a70eecc61d635917de46bea4e6 - HONEYCOMB_DATASET: litmus tests - jobs: - test-backup-restore: - name: "PE ${{ matrix.version }} ${{ matrix.architecture }} on ${{ matrix.image }}" + install-cluster-A: + name: "Cluster A: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" + runs-on: ubuntu-20.04 + env: + BOLT_GEM: true + BOLT_DISABLE_ANALYTICS: true + BUILDEVENT_FILE: '../buildevents.txt' + LANG: 'en_US.UTF-8' + + steps: + - name: "Checkout Source" + uses: actions/checkout@v2 + + - name: "Activate Ruby 2.7" + uses: ruby/setup-ruby@v1 + with: + ruby-version: "2.7" + bundler-cache: true + + - name: "Print bundle environment" + if: ${{ github.repository_owner == 'puppetlabs' }} + run: | + echo ::group::info:bundler + buildevents cmd $TRACE_ID $STEP_ID 'bundle env' -- bundle env + echo ::endgroup:: + + - name: 'Provision test cluster' + timeout-minutes: 15 + run: | + echo ::group::prepare + mkdir -p $HOME/.ssh + echo 'Host *' > $HOME/.ssh/config + echo ' ServerAliveInterval 150' >> $HOME/.ssh/config + echo ' ServerAliveCountMax 2' >> $HOME/.ssh/config + buildevents cmd $TRACE_ID $STEP_ID 'rake spec_prep' -- bundle exec rake spec_prep + echo ::endgroup:: + + echo ::group::provision + buildevents cmd $TRACE_ID $STEP_ID 'bolt plan run peadm_spec::provision_test_cluster' -- \ + bundle exec bolt plan run peadm_spec::provision_test_cluster \ + --modulepath spec/fixtures/modules \ + provider=provision_service \ + image=${{ inputs.image }} \ + architecture=${{ inputs.architecture }} + echo ::endgroup:: + + echo ::group::info:request + cat request.json || true; echo + echo ::endgroup:: + + echo ::group::info:inventory + sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true + echo ::endgroup:: + + - name: Save inventory file A to an artifact + uses: actions/upload-artifact@v2 + with: + name: inventory_A + path: spec/fixtures/litmus_inventory.yaml + + - name: 'Install PE on test cluster' + timeout-minutes: 120 + run: | + buildevents cmd $TRACE_ID $STEP_ID 'bolt plan run peadm_spec::install_test_cluster' -- \ + bundle exec bolt plan run peadm_spec::install_test_cluster \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules \ + architecture=${{ inputs.architecture }} \ + version=${{ inputs.version }} + + install-cluster-B: + name: "Cluster B: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" runs-on: ubuntu-20.04 env: BOLT_GEM: true BOLT_DISABLE_ANALYTICS: true BUILDEVENT_FILE: '../buildevents.txt' LANG: 'en_US.UTF-8' - strategy: - fail-fast: false - matrix: - architecture: - - "${{ github.event.inputs.architecture }}" - version: - - "${{ github.event.inputs.version }}" - image: - - "${{ github.event.inputs.image }}" steps: - name: 'Start SSH session' @@ -52,18 +109,6 @@ jobs: NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} SSH_PASS: ${{ secrets.SSH_PASS }} - - name: "Honeycomb: Start recording" - uses: puppetlabs/kvrhdn-gha-buildevents@pdk-templates-v1 - with: - apikey: ${{ env.HONEYCOMB_WRITEKEY }} - dataset: ${{ env.HONEYCOMB_DATASET }} - job-status: ${{ job.status }} - - - name: "Honeycomb: Start first step" - run: | - echo STEP_ID=setup-test-cluster >> $GITHUB_ENV - echo STEP_START=$(date +%s) >> $GITHUB_ENV - - name: "Checkout Source" uses: actions/checkout@v2 @@ -80,15 +125,6 @@ jobs: buildevents cmd $TRACE_ID $STEP_ID 'bundle env' -- bundle env echo ::endgroup:: - - name: "Honeycomb: Record environment setup time" - if: ${{ always() }} - run: | - echo ::group::honeycomb - buildevents step $TRACE_ID $STEP_ID $STEP_START 'Set up environment' - echo STEP_ID=${{ matrix.architecture }}-${{ matrix.image }}-provision >> $GITHUB_ENV - echo STEP_START=$(date +%s) >> $GITHUB_ENV - echo ::endgroup:: - - name: 'Provision test cluster' timeout-minutes: 15 run: | @@ -105,8 +141,8 @@ jobs: bundle exec bolt plan run peadm_spec::provision_test_cluster \ --modulepath spec/fixtures/modules \ provider=provision_service \ - image=${{ matrix.image }} \ - architecture=${{ matrix.architecture }} + image=${{ inputs.image }} \ + architecture=${{ inputs.architecture }} echo ::endgroup:: echo ::group::info:request @@ -117,15 +153,12 @@ jobs: sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true echo ::endgroup:: - - name: "Honeycomb: Record provision time" - if: ${{ always() }} - run: | - echo ::group::honeycomb - buildevents step $TRACE_ID $STEP_ID $STEP_START 'Provision test cluster' - echo STEP_ID=${{ matrix.architecture }}-${{ matrix.image }}-install >> $GITHUB_ENV - echo STEP_START=$(date +%s) >> $GITHUB_ENV - echo ::endgroup:: - + - name: Save inventory file B to an artifact + uses: actions/upload-artifact@v2 + with: + name: inventory_A + path: spec/fixtures/litmus_inventory.yaml + - name: 'Install PE on test cluster' timeout-minutes: 120 run: | @@ -133,28 +166,57 @@ jobs: bundle exec bolt plan run peadm_spec::install_test_cluster \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ - architecture=${{ matrix.architecture }} \ - version=${{ matrix.version }} + architecture=${{ inputs.architecture }} \ + version=${{ inputs.version }} + + test-backup-restore: + runs-on: ubuntu-20.04 + env: + BOLT_GEM: true + BOLT_DISABLE_ANALYTICS: true + BUILDEVENT_FILE: '../buildevents.txt' + LANG: 'en_US.UTF-8' + needs: [install-cluster-A, install-cluster-B] - - name: "Honeycomb: Record install time" - if: ${{ always() }} + steps: + - name: 'Start SSH session' + if: ${{ github.event.inputs.ssh-debugging == 'true' }} + uses: luchihoratiu/debug-via-ssh@main + with: + NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} + SSH_PASS: ${{ secrets.SSH_PASS }} + + - name: "Checkout Source" + uses: actions/checkout@v2 + + - name: "Activate Ruby 2.7" + uses: ruby/setup-ruby@v1 + with: + ruby-version: "2.7" + bundler-cache: true + + - name: Download artifacts + uses: actions/download-artifact@v3 + with: + path: spec/fixtures/ + + - name: perform PE backup of cluster A + timeout-minutes: 120 run: | - echo ::group::honeycomb - buildevents step $TRACE_ID $STEP_ID $STEP_START 'Install PE on test cluster' - echo STEP_ID=${{ matrix.architecture }}-${{ matrix.image }}-tear_down >> $GITHUB_ENV - echo STEP_START=$(date +%s) >> $GITHUB_ENV - echo ::endgroup:: + buildevents cmd $TRACE_ID $STEP_ID 'bolt plan run peadm_spec::test_backup' -- \ + bundle exec bolt plan run peadm_spec::test_backup \ + --inventoryfile spec/fixtures/inventory_A/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules - - name: 'Wait as long as the file ${HOME}/pause file is present' - if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} + - name: Wait while the pause file is present + if: ${{ github.event.inputs.ssh-debugging == 'true' }} run: | - while [ -f "${HOME}/pause" ] ; do - echo "${HOME}/pause present, sleeping for 60 seconds..." - sleep 60 - done - echo "${HOME}/pause absent, continuing workflow." + while [ -f pause ]; do + echo "Waiting for pause file to be removed" + sleep 10 + done - - name: 'Tear down test cluster' + - name: 'Tear down cluster A' if: ${{ always() }} continue-on-error: true run: | @@ -168,9 +230,17 @@ jobs: echo ::endgroup:: fi - - name: "Honeycomb: Record tear down time" + - name: 'Tear down cluster B' if: ${{ always() }} + continue-on-error: true run: | - echo ::group::honeycomb - buildevents step $TRACE_ID $STEP_ID $STEP_START 'Tear down test cluster' - echo ::endgroup:: + if [ -f spec/fixtures/litmus_inventory.yaml ]; then + echo ::group::tear_down + buildevents cmd $TRACE_ID $STEP_ID 'rake litmus:tear_down' -- bundle exec rake 'litmus:tear_down' + echo ::endgroup:: + + echo ::group::info:request + cat request.json || true; echo + echo ::endgroup:: + fi + diff --git a/spec/acceptance/peadm_spec/plans/test_backup.pp b/spec/acceptance/peadm_spec/plans/test_backup.pp new file mode 100644 index 00000000..c52ccfb8 --- /dev/null +++ b/spec/acceptance/peadm_spec/plans/test_backup.pp @@ -0,0 +1,24 @@ +plan peadm_spec::test_backup() { + $t = get_targets('*') + wait_until_available($t) + + parallelize($t) |$target| { + $fqdn = run_command('hostname -f', $target) + $target.set_var('certname', $fqdn.first['stdout'].chomp) + } + + # run infra status on the primary + $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] + out::verbose("Running peadm::status on primary host ${primary_host}") + run_plan('peadm::status', $primary_host) + + $backup_options = { + 'orchestrator' => true, + 'puppetdb' => true, + 'rbac' => true, + 'activity' => true, + 'ca' => true, + 'classifier' => true, + } + run_plan('peadm::backup', $primary_host, { 'output_directory' => '/tmp', 'backup' => $backup_options }) +} From 05c5009cf220df46b81581be4fa8b7ff86e654ae Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Wed, 12 Apr 2023 15:15:34 +0200 Subject: [PATCH 003/110] remove buildevents refs --- .github/workflows/test-backup-restore.yaml | 24 ++++++++-------------- 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 922a00c6..758cf972 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -28,7 +28,6 @@ jobs: env: BOLT_GEM: true BOLT_DISABLE_ANALYTICS: true - BUILDEVENT_FILE: '../buildevents.txt' LANG: 'en_US.UTF-8' steps: @@ -45,7 +44,7 @@ jobs: if: ${{ github.repository_owner == 'puppetlabs' }} run: | echo ::group::info:bundler - buildevents cmd $TRACE_ID $STEP_ID 'bundle env' -- bundle env + bundle env echo ::endgroup:: - name: 'Provision test cluster' @@ -56,11 +55,10 @@ jobs: echo 'Host *' > $HOME/.ssh/config echo ' ServerAliveInterval 150' >> $HOME/.ssh/config echo ' ServerAliveCountMax 2' >> $HOME/.ssh/config - buildevents cmd $TRACE_ID $STEP_ID 'rake spec_prep' -- bundle exec rake spec_prep + bundle exec rake spec_prep echo ::endgroup:: echo ::group::provision - buildevents cmd $TRACE_ID $STEP_ID 'bolt plan run peadm_spec::provision_test_cluster' -- \ bundle exec bolt plan run peadm_spec::provision_test_cluster \ --modulepath spec/fixtures/modules \ provider=provision_service \ @@ -85,7 +83,6 @@ jobs: - name: 'Install PE on test cluster' timeout-minutes: 120 run: | - buildevents cmd $TRACE_ID $STEP_ID 'bolt plan run peadm_spec::install_test_cluster' -- \ bundle exec bolt plan run peadm_spec::install_test_cluster \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ @@ -98,7 +95,6 @@ jobs: env: BOLT_GEM: true BOLT_DISABLE_ANALYTICS: true - BUILDEVENT_FILE: '../buildevents.txt' LANG: 'en_US.UTF-8' steps: @@ -122,7 +118,7 @@ jobs: if: ${{ github.repository_owner == 'puppetlabs' }} run: | echo ::group::info:bundler - buildevents cmd $TRACE_ID $STEP_ID 'bundle env' -- bundle env + bundle env echo ::endgroup:: - name: 'Provision test cluster' @@ -133,11 +129,10 @@ jobs: echo 'Host *' > $HOME/.ssh/config echo ' ServerAliveInterval 150' >> $HOME/.ssh/config echo ' ServerAliveCountMax 2' >> $HOME/.ssh/config - buildevents cmd $TRACE_ID $STEP_ID 'rake spec_prep' -- bundle exec rake spec_prep + bundle exec rake spec_prep echo ::endgroup:: echo ::group::provision - buildevents cmd $TRACE_ID $STEP_ID 'bolt plan run peadm_spec::provision_test_cluster' -- \ bundle exec bolt plan run peadm_spec::provision_test_cluster \ --modulepath spec/fixtures/modules \ provider=provision_service \ @@ -162,7 +157,6 @@ jobs: - name: 'Install PE on test cluster' timeout-minutes: 120 run: | - buildevents cmd $TRACE_ID $STEP_ID 'bolt plan run peadm_spec::install_test_cluster' -- \ bundle exec bolt plan run peadm_spec::install_test_cluster \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ @@ -174,7 +168,6 @@ jobs: env: BOLT_GEM: true BOLT_DISABLE_ANALYTICS: true - BUILDEVENT_FILE: '../buildevents.txt' LANG: 'en_US.UTF-8' needs: [install-cluster-A, install-cluster-B] @@ -203,7 +196,6 @@ jobs: - name: perform PE backup of cluster A timeout-minutes: 120 run: | - buildevents cmd $TRACE_ID $STEP_ID 'bolt plan run peadm_spec::test_backup' -- \ bundle exec bolt plan run peadm_spec::test_backup \ --inventoryfile spec/fixtures/inventory_A/litmus_inventory.yaml \ --modulepath spec/fixtures/modules @@ -220,9 +212,9 @@ jobs: if: ${{ always() }} continue-on-error: true run: | - if [ -f spec/fixtures/litmus_inventory.yaml ]; then + if [ -f spec/fixtures/inventory_A/litmus_inventory.yaml ]; then echo ::group::tear_down - buildevents cmd $TRACE_ID $STEP_ID 'rake litmus:tear_down' -- bundle exec rake 'litmus:tear_down' + bundle exec rake 'litmus:tear_down' echo ::endgroup:: echo ::group::info:request @@ -234,9 +226,9 @@ jobs: if: ${{ always() }} continue-on-error: true run: | - if [ -f spec/fixtures/litmus_inventory.yaml ]; then + if [ -f spec/fixtures/inventory_B/litmus_inventory.yaml ]; then echo ::group::tear_down - buildevents cmd $TRACE_ID $STEP_ID 'rake litmus:tear_down' -- bundle exec rake 'litmus:tear_down' + bundle exec rake 'litmus:tear_down' echo ::endgroup:: echo ::group::info:request From 42ee72feceab42c4c327a462a02e3b2603de4174 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 14 Apr 2023 17:49:00 +0200 Subject: [PATCH 004/110] copy inventory file for teardown --- .github/workflows/test-backup-restore.yaml | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 758cf972..d9475061 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -75,7 +75,7 @@ jobs: echo ::endgroup:: - name: Save inventory file A to an artifact - uses: actions/upload-artifact@v2 + uses: actions/upload-artifact@v3 with: name: inventory_A path: spec/fixtures/litmus_inventory.yaml @@ -149,7 +149,7 @@ jobs: echo ::endgroup:: - name: Save inventory file B to an artifact - uses: actions/upload-artifact@v2 + uses: actions/upload-artifact@v3 with: name: inventory_A path: spec/fixtures/litmus_inventory.yaml @@ -194,11 +194,21 @@ jobs: path: spec/fixtures/ - name: perform PE backup of cluster A - timeout-minutes: 120 + timeout-minutes: 10 run: | + echo ::group::prepare + mkdir -p $HOME/.ssh + echo 'Host *' > $HOME/.ssh/config + echo ' ServerAliveInterval 150' >> $HOME/.ssh/config + echo ' ServerAliveCountMax 2' >> $HOME/.ssh/config + bundle exec rake spec_prep + echo ::endgroup:: + + echo ::group::backup bundle exec bolt plan run peadm_spec::test_backup \ --inventoryfile spec/fixtures/inventory_A/litmus_inventory.yaml \ --modulepath spec/fixtures/modules + echo ::endgroup:: - name: Wait while the pause file is present if: ${{ github.event.inputs.ssh-debugging == 'true' }} @@ -212,7 +222,8 @@ jobs: if: ${{ always() }} continue-on-error: true run: | - if [ -f spec/fixtures/inventory_A/litmus_inventory.yaml ]; then + cp spec/fixtures/inventory_A/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true + if [ -f spec/fixtures/litmus_inventory.yaml ]; then echo ::group::tear_down bundle exec rake 'litmus:tear_down' echo ::endgroup:: @@ -226,7 +237,8 @@ jobs: if: ${{ always() }} continue-on-error: true run: | - if [ -f spec/fixtures/inventory_B/litmus_inventory.yaml ]; then + cp spec/fixtures/inventory_B/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true + if [ -f spec/fixtures/litmus_inventory.yaml ]; then echo ::group::tear_down bundle exec rake 'litmus:tear_down' echo ::endgroup:: From af03f372adcf9f90d796daf6997b04b06c35c9fc Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sun, 16 Apr 2023 16:00:47 +0200 Subject: [PATCH 005/110] fix typo --- .github/workflows/test-backup-restore.yaml | 2 +- .ruby-version | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 .ruby-version diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index d9475061..ca135708 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -151,7 +151,7 @@ jobs: - name: Save inventory file B to an artifact uses: actions/upload-artifact@v3 with: - name: inventory_A + name: inventory_B path: spec/fixtures/litmus_inventory.yaml - name: 'Install PE on test cluster' diff --git a/.ruby-version b/.ruby-version new file mode 100644 index 00000000..6a81b4c8 --- /dev/null +++ b/.ruby-version @@ -0,0 +1 @@ +2.7.8 From d1345380711ea3ba5860c8c06a88bb8b72112358 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sun, 16 Apr 2023 18:55:16 +0200 Subject: [PATCH 006/110] don't start ssh session during provisioning --- .github/workflows/test-backup-restore.yaml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index ca135708..da5d206c 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -98,13 +98,6 @@ jobs: LANG: 'en_US.UTF-8' steps: - - name: 'Start SSH session' - if: ${{ github.event.inputs.ssh-debugging == 'true' }} - uses: luchihoratiu/debug-via-ssh@main - with: - NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} - SSH_PASS: ${{ secrets.SSH_PASS }} - - name: "Checkout Source" uses: actions/checkout@v2 From b39cec0b46de9ab9613d085ae0a3a406df1eb7e6 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sun, 16 Apr 2023 19:17:42 +0200 Subject: [PATCH 007/110] cleanup even if cancelled --- .github/workflows/test-backup-restore.yaml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index da5d206c..bef7a482 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -162,6 +162,7 @@ jobs: BOLT_GEM: true BOLT_DISABLE_ANALYTICS: true LANG: 'en_US.UTF-8' + if: ${{ always() }} needs: [install-cluster-A, install-cluster-B] steps: @@ -173,20 +174,24 @@ jobs: SSH_PASS: ${{ secrets.SSH_PASS }} - name: "Checkout Source" + if: ${{ always() }} uses: actions/checkout@v2 - name: "Activate Ruby 2.7" + if: ${{ always() }} uses: ruby/setup-ruby@v1 with: ruby-version: "2.7" bundler-cache: true - name: Download artifacts + if: ${{ always() }} uses: actions/download-artifact@v3 with: path: spec/fixtures/ - name: perform PE backup of cluster A + if: needs.install-cluster-A.result == 'success' && needs.install-cluster-B.result == 'success' timeout-minutes: 10 run: | echo ::group::prepare From a0414da70bf957430d85b78b6c1b8a48524e2002 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sun, 16 Apr 2023 19:46:00 +0200 Subject: [PATCH 008/110] wait for pause even if previous step failed --- .github/workflows/test-backup-restore.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index bef7a482..f5ae38ba 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -209,7 +209,7 @@ jobs: echo ::endgroup:: - name: Wait while the pause file is present - if: ${{ github.event.inputs.ssh-debugging == 'true' }} + if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} run: | while [ -f pause ]; do echo "Waiting for pause file to be removed" From 791a2db06621afa06dddafab1d8a3e436e770dda Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sun, 16 Apr 2023 20:47:16 +0200 Subject: [PATCH 009/110] try to wait before tearing down --- .github/workflows/test-backup-restore.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index f5ae38ba..aa011713 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -217,7 +217,7 @@ jobs: done - name: 'Tear down cluster A' - if: ${{ always() }} + # if: ${{ always() }} continue-on-error: true run: | cp spec/fixtures/inventory_A/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true @@ -232,7 +232,7 @@ jobs: fi - name: 'Tear down cluster B' - if: ${{ always() }} + # if: ${{ always() }} continue-on-error: true run: | cp spec/fixtures/inventory_B/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true From ca063d683bfeb21715836bca786ff2c3f97d4a6a Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 17 Apr 2023 15:29:04 +0200 Subject: [PATCH 010/110] add continue on error --- .github/workflows/test-backup-restore.yaml | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index aa011713..60829b73 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -164,6 +164,7 @@ jobs: LANG: 'en_US.UTF-8' if: ${{ always() }} needs: [install-cluster-A, install-cluster-B] + continue-on-error: true steps: - name: 'Start SSH session' @@ -174,24 +175,24 @@ jobs: SSH_PASS: ${{ secrets.SSH_PASS }} - name: "Checkout Source" - if: ${{ always() }} + # if: ${{ always() }} uses: actions/checkout@v2 - name: "Activate Ruby 2.7" - if: ${{ always() }} + # if: ${{ always() }} uses: ruby/setup-ruby@v1 with: ruby-version: "2.7" bundler-cache: true - name: Download artifacts - if: ${{ always() }} + # if: ${{ always() }} uses: actions/download-artifact@v3 with: path: spec/fixtures/ - name: perform PE backup of cluster A - if: needs.install-cluster-A.result == 'success' && needs.install-cluster-B.result == 'success' + # if: needs.install-cluster-A.result == 'success' && needs.install-cluster-B.result == 'success' timeout-minutes: 10 run: | echo ::group::prepare @@ -209,7 +210,8 @@ jobs: echo ::endgroup:: - name: Wait while the pause file is present - if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} + # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} + if: github.event.inputs.ssh-debugging == 'true' run: | while [ -f pause ]; do echo "Waiting for pause file to be removed" From 34311d4416cc490f50ab36e2589007c8b28b675a Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 17 Apr 2023 16:48:19 +0200 Subject: [PATCH 011/110] another attempt --- .github/workflows/test-backup-restore.yaml | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 60829b73..b676217b 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -162,37 +162,37 @@ jobs: BOLT_GEM: true BOLT_DISABLE_ANALYTICS: true LANG: 'en_US.UTF-8' - if: ${{ always() }} + if: always() needs: [install-cluster-A, install-cluster-B] continue-on-error: true steps: - name: 'Start SSH session' - if: ${{ github.event.inputs.ssh-debugging == 'true' }} + if: github.event.inputs.ssh-debugging == 'true' uses: luchihoratiu/debug-via-ssh@main with: NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} SSH_PASS: ${{ secrets.SSH_PASS }} - name: "Checkout Source" - # if: ${{ always() }} + # if: always() uses: actions/checkout@v2 - name: "Activate Ruby 2.7" - # if: ${{ always() }} + # if: always() uses: ruby/setup-ruby@v1 with: ruby-version: "2.7" bundler-cache: true - name: Download artifacts - # if: ${{ always() }} + # if: always() uses: actions/download-artifact@v3 with: path: spec/fixtures/ - name: perform PE backup of cluster A - # if: needs.install-cluster-A.result == 'success' && needs.install-cluster-B.result == 'success' + if: needs.install-cluster-A.result == 'success' && needs.install-cluster-B.result == 'success' timeout-minutes: 10 run: | echo ::group::prepare @@ -210,8 +210,8 @@ jobs: echo ::endgroup:: - name: Wait while the pause file is present - # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} - if: github.event.inputs.ssh-debugging == 'true' + if: always() && github.event.inputs.ssh-debugging == 'true' + # if: github.event.inputs.ssh-debugging == 'true' run: | while [ -f pause ]; do echo "Waiting for pause file to be removed" @@ -219,7 +219,7 @@ jobs: done - name: 'Tear down cluster A' - # if: ${{ always() }} + if: always() continue-on-error: true run: | cp spec/fixtures/inventory_A/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true @@ -234,7 +234,7 @@ jobs: fi - name: 'Tear down cluster B' - # if: ${{ always() }} + if: always() continue-on-error: true run: | cp spec/fixtures/inventory_B/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true From 2db76d52d6e8562acd7e1671ae417aa8c2a82eaa Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Wed, 19 Apr 2023 12:58:31 +0200 Subject: [PATCH 012/110] removed always() --- .github/workflows/test-backup-restore.yaml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index b676217b..65f59a0d 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -210,8 +210,8 @@ jobs: echo ::endgroup:: - name: Wait while the pause file is present - if: always() && github.event.inputs.ssh-debugging == 'true' - # if: github.event.inputs.ssh-debugging == 'true' + # if: always() && github.event.inputs.ssh-debugging == 'true' + if: github.event.inputs.ssh-debugging == 'true' run: | while [ -f pause ]; do echo "Waiting for pause file to be removed" @@ -219,8 +219,8 @@ jobs: done - name: 'Tear down cluster A' - if: always() - continue-on-error: true + # if: always() + # continue-on-error: true run: | cp spec/fixtures/inventory_A/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true if [ -f spec/fixtures/litmus_inventory.yaml ]; then @@ -234,8 +234,8 @@ jobs: fi - name: 'Tear down cluster B' - if: always() - continue-on-error: true + # if: always() + # continue-on-error: true run: | cp spec/fixtures/inventory_B/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true if [ -f spec/fixtures/litmus_inventory.yaml ]; then From eb86c0376b4e424214abffd9e4727b863cbd0700 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Wed, 19 Apr 2023 13:41:53 +0200 Subject: [PATCH 013/110] don't wait in the ssh loop if previous job was cancelled --- .github/workflows/test-backup-restore.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 65f59a0d..07e83320 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -168,7 +168,7 @@ jobs: steps: - name: 'Start SSH session' - if: github.event.inputs.ssh-debugging == 'true' + if: github.event.inputs.ssh-debugging == 'true' && needs.install-cluster-A.result != 'cancelled' && needs.install-cluster-B.result != 'cancelled' uses: luchihoratiu/debug-via-ssh@main with: NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} From 293c5a4846b1a47032f2a79fc52ed5a4354dd8da Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Wed, 19 Apr 2023 16:03:11 +0200 Subject: [PATCH 014/110] continue on error --- .github/workflows/test-backup-restore.yaml | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 07e83320..e92297ca 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -168,6 +168,7 @@ jobs: steps: - name: 'Start SSH session' + continue-on-error: true if: github.event.inputs.ssh-debugging == 'true' && needs.install-cluster-A.result != 'cancelled' && needs.install-cluster-B.result != 'cancelled' uses: luchihoratiu/debug-via-ssh@main with: @@ -194,6 +195,7 @@ jobs: - name: perform PE backup of cluster A if: needs.install-cluster-A.result == 'success' && needs.install-cluster-B.result == 'success' timeout-minutes: 10 + continue-on-error: true run: | echo ::group::prepare mkdir -p $HOME/.ssh @@ -210,7 +212,8 @@ jobs: echo ::endgroup:: - name: Wait while the pause file is present - # if: always() && github.event.inputs.ssh-debugging == 'true' + continue-on-error: true + # if: always() && github.event.inputs.ssh-debugging == 'true' if: github.event.inputs.ssh-debugging == 'true' run: | while [ -f pause ]; do @@ -220,7 +223,7 @@ jobs: - name: 'Tear down cluster A' # if: always() - # continue-on-error: true + continue-on-error: true run: | cp spec/fixtures/inventory_A/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true if [ -f spec/fixtures/litmus_inventory.yaml ]; then From fc5d215c9ecb8c5753074a70128253d40cd49c07 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Wed, 19 Apr 2023 21:15:07 +0200 Subject: [PATCH 015/110] now looking at correct pause file --- .github/workflows/test-backup-restore.yaml | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index e92297ca..95584c68 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -211,16 +211,17 @@ jobs: --modulepath spec/fixtures/modules echo ::endgroup:: - - name: Wait while the pause file is present + - name: 'Wait as long as the file ${HOME}/pause file is present' continue-on-error: true - # if: always() && github.event.inputs.ssh-debugging == 'true' + # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} if: github.event.inputs.ssh-debugging == 'true' run: | - while [ -f pause ]; do - echo "Waiting for pause file to be removed" + while [ -f "${HOME}/pause" ] ; do + echo "${HOME}/pause present, sleeping for 60 seconds..." sleep 10 - done - + done + echo "${HOME}/pause absent, continuing workflow." + - name: 'Tear down cluster A' # if: always() continue-on-error: true From d3bcb83e9e1ab476f7064881066dfd19d31bc3ac Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 24 Apr 2023 18:35:58 +0200 Subject: [PATCH 016/110] refactor into 2 jobs --- .github/workflows/test-backup-restore.yaml | 177 ++++++++++----------- 1 file changed, 80 insertions(+), 97 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 95584c68..20c901cb 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -22,8 +22,8 @@ on: default: 'false' jobs: - install-cluster-A: - name: "Cluster A: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" + backup: + name: "Backup: Cluster A: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" runs-on: ubuntu-20.04 env: BOLT_GEM: true @@ -74,11 +74,11 @@ jobs: sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true echo ::endgroup:: - - name: Save inventory file A to an artifact - uses: actions/upload-artifact@v3 - with: - name: inventory_A - path: spec/fixtures/litmus_inventory.yaml + # - name: Save inventory file A to an artifact + # uses: actions/upload-artifact@v3 + # with: + # name: inventory_A + # path: spec/fixtures/litmus_inventory.yaml - name: 'Install PE on test cluster' timeout-minutes: 120 @@ -88,9 +88,66 @@ jobs: --modulepath spec/fixtures/modules \ architecture=${{ inputs.architecture }} \ version=${{ inputs.version }} + - name: 'Start SSH session' + continue-on-error: true + if: github.event.inputs.ssh-debugging == 'true' && needs.install-cluster-A.result != 'cancelled' && needs.install-cluster-B.result != 'cancelled' + uses: luchihoratiu/debug-via-ssh@main + with: + NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} + SSH_PASS: ${{ secrets.SSH_PASS }} + + # - name: Download artifacts + # # if: always() + # uses: actions/download-artifact@v3 + # with: + # path: spec/fixtures/ + + - name: perform PE backup of cluster A + if: needs.install-cluster-A.result == 'success' && needs.install-cluster-B.result == 'success' + timeout-minutes: 10 + continue-on-error: true + run: | + echo ::group::prepare + mkdir -p $HOME/.ssh + echo 'Host *' > $HOME/.ssh/config + echo ' ServerAliveInterval 150' >> $HOME/.ssh/config + echo ' ServerAliveCountMax 2' >> $HOME/.ssh/config + bundle exec rake spec_prep + echo ::endgroup:: - install-cluster-B: - name: "Cluster B: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" + echo ::group::backup + bundle exec bolt plan run peadm_spec::test_backup \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules + echo ::endgroup:: + + - name: 'Wait as long as the file ${HOME}/pause file is present' + continue-on-error: true + # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} + if: github.event.inputs.ssh-debugging == 'true' + run: | + while [ -f "${HOME}/pause" ] ; do + echo "${HOME}/pause present, sleeping for 60 seconds..." + sleep 10 + done + echo "${HOME}/pause absent, continuing workflow." + + - name: 'Tear down cluster A' + # if: always() + continue-on-error: true + run: | + if [ -f spec/fixtures/litmus_inventory.yaml ]; then + echo ::group::tear_down + bundle exec rake 'litmus:tear_down' + echo ::endgroup:: + + echo ::group::info:request + cat request.json || true; echo + echo ::endgroup:: + fi + + restore: + name: "Restore: Cluster B: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" runs-on: ubuntu-20.04 env: BOLT_GEM: true @@ -141,11 +198,11 @@ jobs: sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true echo ::endgroup:: - - name: Save inventory file B to an artifact - uses: actions/upload-artifact@v3 - with: - name: inventory_B - path: spec/fixtures/litmus_inventory.yaml + # - name: Save inventory file B to an artifact + # uses: actions/upload-artifact@v3 + # with: + # name: inventory_B + # path: spec/fixtures/litmus_inventory.yaml - name: 'Install PE on test cluster' timeout-minutes: 120 @@ -155,91 +212,18 @@ jobs: --modulepath spec/fixtures/modules \ architecture=${{ inputs.architecture }} \ version=${{ inputs.version }} - - test-backup-restore: - runs-on: ubuntu-20.04 - env: - BOLT_GEM: true - BOLT_DISABLE_ANALYTICS: true - LANG: 'en_US.UTF-8' - if: always() - needs: [install-cluster-A, install-cluster-B] - continue-on-error: true - steps: - - name: 'Start SSH session' - continue-on-error: true - if: github.event.inputs.ssh-debugging == 'true' && needs.install-cluster-A.result != 'cancelled' && needs.install-cluster-B.result != 'cancelled' - uses: luchihoratiu/debug-via-ssh@main + - name: Wait for backup to finish + uses: lewagon/wait-on-check-action@v1.3.1 with: - NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} - SSH_PASS: ${{ secrets.SSH_PASS }} - - - name: "Checkout Source" - # if: always() - uses: actions/checkout@v2 - - - name: "Activate Ruby 2.7" - # if: always() - uses: ruby/setup-ruby@v1 - with: - ruby-version: "2.7" - bundler-cache: true - - - name: Download artifacts - # if: always() - uses: actions/download-artifact@v3 - with: - path: spec/fixtures/ - - - name: perform PE backup of cluster A - if: needs.install-cluster-A.result == 'success' && needs.install-cluster-B.result == 'success' - timeout-minutes: 10 - continue-on-error: true - run: | - echo ::group::prepare - mkdir -p $HOME/.ssh - echo 'Host *' > $HOME/.ssh/config - echo ' ServerAliveInterval 150' >> $HOME/.ssh/config - echo ' ServerAliveCountMax 2' >> $HOME/.ssh/config - bundle exec rake spec_prep - echo ::endgroup:: - - echo ::group::backup - bundle exec bolt plan run peadm_spec::test_backup \ - --inventoryfile spec/fixtures/inventory_A/litmus_inventory.yaml \ - --modulepath spec/fixtures/modules - echo ::endgroup:: - - - name: 'Wait as long as the file ${HOME}/pause file is present' - continue-on-error: true - # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} - if: github.event.inputs.ssh-debugging == 'true' - run: | - while [ -f "${HOME}/pause" ] ; do - echo "${HOME}/pause present, sleeping for 60 seconds..." - sleep 10 - done - echo "${HOME}/pause absent, continuing workflow." - - - name: 'Tear down cluster A' - # if: always() - continue-on-error: true - run: | - cp spec/fixtures/inventory_A/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true - if [ -f spec/fixtures/litmus_inventory.yaml ]; then - echo ::group::tear_down - bundle exec rake 'litmus:tear_down' - echo ::endgroup:: - - echo ::group::info:request - cat request.json || true; echo - echo ::endgroup:: - fi - + ref: ${{ github.ref }} + check-name: 'Wait as long as the file ${HOME}/pause file is present' + repo-token: ${{ secrets.GITHUB_TOKEN }} + wait-interval: 10 + - name: 'Tear down cluster B' - # if: always() - # continue-on-error: true + # if: always() + # continue-on-error: true run: | cp spec/fixtures/inventory_B/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true if [ -f spec/fixtures/litmus_inventory.yaml ]; then @@ -251,4 +235,3 @@ jobs: cat request.json || true; echo echo ::endgroup:: fi - From 1a9deb842bf301f810cbfa89edda6e3fb3d1a46c Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 24 Apr 2023 19:08:30 +0200 Subject: [PATCH 017/110] fix check wait; fix indentation --- .github/workflows/test-backup-restore.yaml | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 20c901cb..b9f999ba 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -88,13 +88,14 @@ jobs: --modulepath spec/fixtures/modules \ architecture=${{ inputs.architecture }} \ version=${{ inputs.version }} - - name: 'Start SSH session' - continue-on-error: true - if: github.event.inputs.ssh-debugging == 'true' && needs.install-cluster-A.result != 'cancelled' && needs.install-cluster-B.result != 'cancelled' - uses: luchihoratiu/debug-via-ssh@main - with: - NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} - SSH_PASS: ${{ secrets.SSH_PASS }} + + - name: 'Start SSH session' + continue-on-error: true + if: github.event.inputs.ssh-debugging == 'true' && needs.install-cluster-A.result != 'cancelled' && needs.install-cluster-B.result != 'cancelled' + uses: luchihoratiu/debug-via-ssh@main + with: + NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} + SSH_PASS: ${{ secrets.SSH_PASS }} # - name: Download artifacts # # if: always() @@ -217,7 +218,7 @@ jobs: uses: lewagon/wait-on-check-action@v1.3.1 with: ref: ${{ github.ref }} - check-name: 'Wait as long as the file ${HOME}/pause file is present' + check-name: "Backup: Cluster A: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" repo-token: ${{ secrets.GITHUB_TOKEN }} wait-interval: 10 From 8d50129c4f43a40b4b9edf1180c8bd5651fa1aa9 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 24 Apr 2023 19:45:17 +0200 Subject: [PATCH 018/110] fix conditions --- .github/workflows/test-backup-restore.yaml | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index b9f999ba..dada33f6 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -90,8 +90,7 @@ jobs: version=${{ inputs.version }} - name: 'Start SSH session' - continue-on-error: true - if: github.event.inputs.ssh-debugging == 'true' && needs.install-cluster-A.result != 'cancelled' && needs.install-cluster-B.result != 'cancelled' + if: github.event.inputs.ssh-debugging == 'true' uses: luchihoratiu/debug-via-ssh@main with: NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} @@ -104,7 +103,6 @@ jobs: # path: spec/fixtures/ - name: perform PE backup of cluster A - if: needs.install-cluster-A.result == 'success' && needs.install-cluster-B.result == 'success' timeout-minutes: 10 continue-on-error: true run: | @@ -134,8 +132,7 @@ jobs: echo "${HOME}/pause absent, continuing workflow." - name: 'Tear down cluster A' - # if: always() - continue-on-error: true + if: always() run: | if [ -f spec/fixtures/litmus_inventory.yaml ]; then echo ::group::tear_down @@ -223,8 +220,7 @@ jobs: wait-interval: 10 - name: 'Tear down cluster B' - # if: always() - # continue-on-error: true + if: always() run: | cp spec/fixtures/inventory_B/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true if [ -f spec/fixtures/litmus_inventory.yaml ]; then From 7214920691285ce5386b2cb0f10bb1615cfb0162 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 28 Apr 2023 15:11:01 +0200 Subject: [PATCH 019/110] backup test WIP --- spec/acceptance/peadm_spec/plans/test_backup.pp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/acceptance/peadm_spec/plans/test_backup.pp b/spec/acceptance/peadm_spec/plans/test_backup.pp index c52ccfb8..9c02eeb2 100644 --- a/spec/acceptance/peadm_spec/plans/test_backup.pp +++ b/spec/acceptance/peadm_spec/plans/test_backup.pp @@ -9,7 +9,7 @@ # run infra status on the primary $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] - out::verbose("Running peadm::status on primary host ${primary_host}") + out::message("Running peadm::status on primary host ${primary_host}") run_plan('peadm::status', $primary_host) $backup_options = { From 01b23db7e424d81a62d0b90b2ad166591dd78e0d Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 10 Nov 2023 14:20:09 +0100 Subject: [PATCH 020/110] allow for ca backup --- types/recovery_opts.pp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/types/recovery_opts.pp b/types/recovery_opts.pp index c759e955..03b318db 100644 --- a/types/recovery_opts.pp +++ b/types/recovery_opts.pp @@ -3,6 +3,6 @@ 'puppetdb' => Optional[Boolean], 'rbac' => Optional[Boolean], 'activity' => Optional[Boolean], - 'ca' => Optional[Boolean[false]], + 'ca' => Optional[Boolean], 'classifier' => Optional[Boolean], }] From 0d4d4859cc3672761ba0adf92585d616eda7348e Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 21 Nov 2023 12:32:57 +0100 Subject: [PATCH 021/110] change implementation to ruby --- tasks/transform_classification_groups.json | 28 +++++----- tasks/transform_classification_groups.rb | 62 ++++++++++++++++++++++ 2 files changed, 75 insertions(+), 15 deletions(-) create mode 100644 tasks/transform_classification_groups.rb diff --git a/tasks/transform_classification_groups.json b/tasks/transform_classification_groups.json index 4284d04f..3bb9c203 100644 --- a/tasks/transform_classification_groups.json +++ b/tasks/transform_classification_groups.json @@ -1,17 +1,15 @@ { - "description": "Transform the user groups from a source backup to a list of groups on the target server", - "parameters": { - "source_directory": { - "type": "String", - "description": "Location of Source node group yaml file" - }, - "working_directory": { - "type": "String", - "description": "Location of target node group yaml file and where to create the transformed file" - } + "description": "Transform the user groups from a source backup to a list of groups on the target server", + "parameters": { + "source_directory": { + "type": "String", + "description": "Location of Source node group yaml file" }, - "input_method": "stdin", - "implementations": [ - {"name": "transform_classification_groups.py"} - ] - } \ No newline at end of file + "working_directory": { + "type": "String", + "description": "Location of target node group yaml file and where to create the transformed file" + } + }, + "input_method": "stdin", + "implementations": [{ "name": "transform_classification_groups.rb" }] +} diff --git a/tasks/transform_classification_groups.rb b/tasks/transform_classification_groups.rb new file mode 100644 index 00000000..ab025a40 --- /dev/null +++ b/tasks/transform_classification_groups.rb @@ -0,0 +1,62 @@ +#!/opt/puppetlabs/puppet/bin/ruby +# frozen_string_literal: true + +# This script takes two classification outputs from source and target puppet infrastructure and +# takes the user definitions from the source and adds them to the infrastructure definitions of the +# target. This allows the ability to restore a backup of user node definitions. + +require 'json' + +# Parse JSON from stdin +params = JSON.parse(STDIN.read) +source_classification_file = "#{params['source_directory']}/classification_backup.json" +target_classification_file = "#{params['working_directory']}/classification_backup.json" +transformed_classification_file = "#{params['working_directory']}/transformed_classification.json" + +# Function to remove subgroups +def removesubgroups(data_rsg, id_rsg) + groups = data_rsg.select { |x| x["parent"] == id_rsg } + groups.each do |group| + subid = group["id"] + data_rsg.reject! { |x| x["id"] == subid } + data_rsg = removesubgroups(data_rsg, subid) + end + data_rsg +end + +# Function to add subgroups +def addsubgroups(data_asg, id_asg, peinf_asg) + groups = data_asg.select { |x| x["parent"] == id_asg } + peinf_asg += groups + groups.each do |group| + subid = group["id"] + peinf_asg = addsubgroups(data_asg, subid, peinf_asg) + end + peinf_asg +end + +# Read the backup classification +data = JSON.parse(File.read(source_classification_file)) + +# Read the DR server classification +data_DR = JSON.parse(File.read(target_classification_file)) + +# Find the infrastructure group and its ID +peinf = data.select { |x| x["name"] == "PE Infrastructure" } +group_id = peinf[0]["id"] + +# Remove this group from the list and recursively remove all subgroups +data.reject! { |x| x["id"] == group_id } +data = removesubgroups(data, group_id) + +# Find the DR infrastructure group and its ID +peinf_DR = data_DR.select { |x| x["name"] == "PE Infrastructure" } +id_DR = peinf_DR[0]["id"] + +# Recursively go through inf groups to get the full tree +peinf_DR = addsubgroups(data_DR, id_DR, peinf_DR) + +# Add the contents of the backup classification without PE inf to the DR PE inf groups +# and write to a file +peinf_transformed_groups = data + peinf_DR +File.open(transformed_classification_file, 'w') { |file| file.write(JSON.pretty_generate(peinf_transformed_groups)) } From 9c764db63c5900670c6cb9677fc74aff48ff5b34 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 21 Nov 2023 12:33:12 +0100 Subject: [PATCH 022/110] formatting --- plans/restore_ca.pp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plans/restore_ca.pp b/plans/restore_ca.pp index d5ff84da..82d0a961 100644 --- a/plans/restore_ca.pp +++ b/plans/restore_ca.pp @@ -1,13 +1,13 @@ plan peadm::restore_ca( Peadm::SingleTargetSpec $target, String $file_path, - Optional[String] $recovery_directory = '/tmp/peadm_recovery', + Optional[String] $recovery_directory = '/tmp/peadm_recovery', ) { out::message('# Restoring ca and ssl certificates') # lint:ignore:strict_indent run_command(@("CMD"/L), $target) - /opt/puppetlabs/bin/puppet-backup restore \ + /opt/puppetlabs/bin/puppet-backup restore \ --scope=certs \ --tempdir=${shellquote($recovery_directory)} \ --force \ From 15ad2dfbf0e6f6dd79050614d0e43c7d0a9777eb Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 21 Nov 2023 13:20:06 +0100 Subject: [PATCH 023/110] typo and path fixes, use cert auth for pdb export --- plans/restore.pp | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/plans/restore.pp b/plans/restore.pp index 2bbe7d8c..8b53277c 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -73,7 +73,6 @@ run_task('peadm::backup_classification', $primary_target, directory => $recovery_directory ) - out::message("# Backed up current classification to ${recovery_directory}/classification_backup.json") run_task('peadm::transform_classification_groups', $primary_target, source_directory => "${recovery_directory}/classifier", @@ -81,7 +80,7 @@ ) run_task('peadm::restore_classification', $primary_target, - classification_file => "${recovery_directory}/classification_backup.json", + classification_file => "${recovery_directory}/transformed_classification.json", ) } @@ -93,7 +92,7 @@ --scope=certs \ --tempdir=${shellquote($recovery_directory)} \ --force \ - ${shellquote($recovery_directory)}/classifier/pe_backup-*tgz + ${shellquote($recovery_directory)}/ca/pe_backup-*tgz | CMD } @@ -102,8 +101,12 @@ # TODO: consider adding a heuristic to skip when innappropriate due to size # or other factors. if getvar('recovery_opts.puppetdb') { + out::message('# Exporting puppetdb') run_command(@("CMD"/L), $primary_target) - /opt/puppetlabs/bin/puppet-db export ${shellquote($recovery_directory)}/puppetdb-archive.bin + /opt/puppetlabs/bin/puppet-db export \ + --cert=$(/opt/puppetlabs/bin/puppet config print hostcert) \ + --key=$(/opt/puppetlabs/bin/puppet config print hostprivkey) \ + ${shellquote($recovery_directory)}/puppetdb-archive.bin | CMD } @@ -125,7 +128,7 @@ if getvar('recovery_opts.orchestrator') { out::message('# Restoring orchestrator secret keys') run_command(@("CMD"/L), $primary_target) - cp -rp ${shellquote($recovery_directory)}/orchestrator/secrets/* /etc/puppetlabs/orchestration-services/conf.d/secrets/ + cp -rp ${shellquote($recovery_directory)}/orchestrator/secrets/* /etc/puppetlabs/orchestration-services/conf.d/secrets/ | CMD } # lint:endignore From b12008a0b1bf901d5957ce9b8c3825fe7b321fd8 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 21 Nov 2023 13:45:46 +0100 Subject: [PATCH 024/110] update REFERENCE.md --- REFERENCE.md | 607 ++++++++++++++++++++++++++------------------------- 1 file changed, 305 insertions(+), 302 deletions(-) diff --git a/REFERENCE.md b/REFERENCE.md index f9151023..1f6332d1 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -6,6 +6,9 @@ ### Classes +#### Public Classes + + #### Private Classes * `peadm::setup::convert_node_manager`: Used during the peadm::convert plan @@ -15,37 +18,37 @@ ### Functions -* [`peadm::assert_supported_architecture`](#peadm--assert_supported_architecture): Assert that the architecture given is a supported one -* [`peadm::assert_supported_bolt_version`](#peadm--assert_supported_bolt_version): Assert that the Bolt executable running PEAdm is a supported version -* [`peadm::assert_supported_pe_version`](#peadm--assert_supported_pe_version): Assert that the PE version given is supported by PEAdm -* [`peadm::bolt_version`](#peadm--bolt_version) -* [`peadm::certname`](#peadm--certname): Return the certname of the given target-like input -* [`peadm::check_version_and_known_hosts`](#peadm--check_version_and_known_hosts): Checks PE verison and warns about setting r10k_known_hosts -* [`peadm::convert_hash`](#peadm--convert_hash): converts two arrays into hash -* [`peadm::convert_status`](#peadm--convert_status): Transforms a value in a human readable status with or without colors -* [`peadm::determine_status`](#peadm--determine_status): Produces a summarized hash of the given status data -* [`peadm::fail_on_transport`](#peadm--fail_on_transport): Fails if any nodes have the chosen transport. Useful for excluding PCP when it's not appopriate -* [`peadm::file_content_upload`](#peadm--file_content_upload) -* [`peadm::file_or_content`](#peadm--file_or_content) -* [`peadm::flatten_compact`](#peadm--flatten_compact) -* [`peadm::generate_pe_conf`](#peadm--generate_pe_conf): Generate a pe.conf file in JSON format -* [`peadm::get_pe_conf`](#peadm--get_pe_conf) -* [`peadm::get_targets`](#peadm--get_targets): Accept undef or a SingleTargetSpec, and return an Array[Target, 1, 0]. This differs from get_target() in that: - It returns an Array[Target -* [`peadm::node_manager_yaml_location`](#peadm--node_manager_yaml_location) -* [`peadm::oid`](#peadm--oid) -* [`peadm::plan_step`](#peadm--plan_step) -* [`peadm::recovery_opts_default`](#peadm--recovery_opts_default) -* [`peadm::update_pe_conf`](#peadm--update_pe_conf): Update the pe.conf file on a target with the provided hash -* [`peadm::wait_until_service_ready`](#peadm--wait_until_service_ready): A convenience function to help remember port numbers for services and handle running the wait_until_service_ready task +* [`peadm::assert_supported_architecture`](#peadmassert_supported_architecture): Assert that the architecture given is a supported one +* [`peadm::assert_supported_bolt_version`](#peadmassert_supported_bolt_version): Assert that the Bolt executable running PEAdm is a supported version +* [`peadm::assert_supported_pe_version`](#peadmassert_supported_pe_version): Assert that the PE version given is supported by PEAdm +* [`peadm::bolt_version`](#peadmbolt_version) +* [`peadm::certname`](#peadmcertname): Return the certname of the given target-like input +* [`peadm::check_version_and_known_hosts`](#peadmcheck_version_and_known_hosts): Checks PE verison and warns about setting r10k_known_hosts +* [`peadm::convert_hash`](#peadmconvert_hash): converts two arrays into hash +* [`peadm::convert_status`](#peadmconvert_status): Transforms a value in a human readable status with or without colors +* [`peadm::determine_status`](#peadmdetermine_status): Produces a summarized hash of the given status data +* [`peadm::fail_on_transport`](#peadmfail_on_transport): Fails if any nodes have the chosen transport. Useful for excluding PCP when it's not appopriate +* [`peadm::file_content_upload`](#peadmfile_content_upload) +* [`peadm::file_or_content`](#peadmfile_or_content) +* [`peadm::flatten_compact`](#peadmflatten_compact) +* [`peadm::generate_pe_conf`](#peadmgenerate_pe_conf): Generate a pe.conf file in JSON format +* [`peadm::get_pe_conf`](#peadmget_pe_conf) +* [`peadm::get_targets`](#peadmget_targets): Accept undef or a SingleTargetSpec, and return an Array[Target, 1, 0]. This differs from get_target() in that: - It returns an Array[Target +* [`peadm::node_manager_yaml_location`](#peadmnode_manager_yaml_location) +* [`peadm::oid`](#peadmoid) +* [`peadm::plan_step`](#peadmplan_step) +* [`peadm::recovery_opts_default`](#peadmrecovery_opts_default) +* [`peadm::update_pe_conf`](#peadmupdate_pe_conf): Update the pe.conf file on a target with the provided hash +* [`peadm::wait_until_service_ready`](#peadmwait_until_service_ready): A convenience function to help remember port numbers for services and handle running the wait_until_service_ready task ### Data types -* [`Peadm::Known_hosts`](#Peadm--Known_hosts) -* [`Peadm::Ldap_config`](#Peadm--Ldap_config) -* [`Peadm::Pe_version`](#Peadm--Pe_version) -* [`Peadm::Pem`](#Peadm--Pem) -* [`Peadm::Recovery_opts`](#Peadm--Recovery_opts) -* [`Peadm::SingleTargetSpec`](#Peadm--SingleTargetSpec): A SingleTargetSpec represents any String, Target or single-element array of one or the other that can be passed to get_targets() to return an +* [`Peadm::Known_hosts`](#peadmknown_hosts) +* [`Peadm::Ldap_config`](#peadmldap_config) +* [`Peadm::Pe_version`](#peadmpe_version) +* [`Peadm::Pem`](#peadmpem) +* [`Peadm::Recovery_opts`](#peadmrecovery_opts) +* [`Peadm::SingleTargetSpec`](#peadmsingletargetspec): A SingleTargetSpec represents any String, Target or single-element array of one or the other that can be passed to get_targets() to return an ### Tasks @@ -84,14 +87,14 @@ #### Public Plans -* [`peadm::add_database`](#peadm--add_database) -* [`peadm::backup_ca`](#peadm--backup_ca) -* [`peadm::convert`](#peadm--convert): Convert an existing PE cluster to a PEAdm-managed cluster -* [`peadm::install`](#peadm--install): Install a new PE cluster -* [`peadm::modify_certificate`](#peadm--modify_certificate): Modify the certificate of one or more targets -* [`peadm::restore_ca`](#peadm--restore_ca) -* [`peadm::status`](#peadm--status): Return status information from one or more PE clusters in a table format -* [`peadm::upgrade`](#peadm--upgrade): Upgrade a PEAdm-managed cluster +* [`peadm::add_database`](#peadmadd_database) +* [`peadm::backup_ca`](#peadmbackup_ca) +* [`peadm::convert`](#peadmconvert): Convert an existing PE cluster to a PEAdm-managed cluster +* [`peadm::install`](#peadminstall): Install a new PE cluster +* [`peadm::modify_certificate`](#peadmmodify_certificate): Modify the certificate of one or more targets +* [`peadm::restore_ca`](#peadmrestore_ca) +* [`peadm::status`](#peadmstatus): Return status information from one or more PE clusters in a table format +* [`peadm::upgrade`](#peadmupgrade): Upgrade a PEAdm-managed cluster #### Private Plans @@ -99,8 +102,10 @@ * `peadm::add_replica`: Replace a replica host for a Standard or Large architecture. Supported use cases: 1: The existing replica is broken, we have a fresh new VM we want to provision the replica to. +* `peadm::backup`: Backup the core user settings for puppet infrastructure * `peadm::misc::divert_code_manager`: This plan exists to account for a scenario where a PE XL * `peadm::modify_cert_extensions` +* `peadm::restore`: Restore the core user settings for puppet infrastructure from backup * `peadm::subplans::component_install`: Install a new PEADM component * `peadm::subplans::configure`: Configure first-time classification and DR setup * `peadm::subplans::db_populate`: Destructively (re)populates a new or existing database with the contents or a known good source @@ -118,9 +123,11 @@ Supported use cases: * `peadm::util::update_classification`: Configure classification * `peadm::util::update_db_setting`: Make updates to PuppetDB database settings +## Classes + ## Functions -### `peadm::assert_supported_architecture` +### `peadm::assert_supported_architecture` Type: Puppet Language @@ -162,7 +169,7 @@ Data type: `Variant[TargetSpec, Undef]` -### `peadm::assert_supported_bolt_version` +### `peadm::assert_supported_bolt_version` Type: Puppet Language @@ -178,7 +185,7 @@ Accepts a parameter for the $supported_bolt_version for unit testing purposes Returns: `Struct[{ 'supported' => Boolean }]` -### `peadm::assert_supported_pe_version` +### `peadm::assert_supported_pe_version` Type: Puppet Language @@ -208,7 +215,7 @@ Data type: `Boolean` -### `peadm::bolt_version` +### `peadm::bolt_version` Type: Ruby 4.x API @@ -220,7 +227,7 @@ The peadm::bolt_version function. Returns: `Any` -### `peadm::certname` +### `peadm::certname` Type: Puppet Language @@ -252,21 +259,17 @@ Returns: `Variant[String, Undef]` ##### `target` -Data type: - -```puppet -Variant[Target, +Data type: `Variant[Target, String, Undef, Array[Target,1,1], Array[String,1,1], Array[Undef,1,1], - Array[Any,0,0]] -``` + Array[Any,0,0]]` -### `peadm::check_version_and_known_hosts` +### `peadm::check_version_and_known_hosts` Type: Puppet Language @@ -316,7 +319,7 @@ Data type: `Optional[Peadm::Known_hosts]` -### `peadm::convert_hash` +### `peadm::convert_hash` Type: Puppet Language @@ -362,7 +365,7 @@ Data type: `Array[Array]` data to be merged into an array with the keys -### `peadm::convert_status` +### `peadm::convert_status` Type: Puppet Language @@ -457,7 +460,7 @@ Data type: `Optional[Boolean]` Adds colors to the status, defaults to true -### `peadm::determine_status` +### `peadm::determine_status` Type: Puppet Language @@ -577,7 +580,7 @@ Data type: `Boolean` Adds colors to the status, defaults to true -### `peadm::fail_on_transport` +### `peadm::fail_on_transport` Type: Puppet Language @@ -611,7 +614,7 @@ Data type: `String` -### `peadm::file_content_upload` +### `peadm::file_content_upload` Type: Ruby 4.x API @@ -641,7 +644,7 @@ Data type: `TargetOrTargets` -### `peadm::file_or_content` +### `peadm::file_or_content` Type: Puppet Language @@ -671,7 +674,7 @@ Data type: `Variant[String, Undef]` -### `peadm::flatten_compact` +### `peadm::flatten_compact` Type: Puppet Language @@ -689,7 +692,7 @@ Data type: `Array` -### `peadm::generate_pe_conf` +### `peadm::generate_pe_conf` Type: Puppet Language @@ -708,7 +711,7 @@ Data type: `Hash` A hash of settings to set in the config file. Any keys that are set to undef will not be included in the config file. -### `peadm::get_pe_conf` +### `peadm::get_pe_conf` Type: Puppet Language @@ -726,7 +729,7 @@ Data type: `Target` -### `peadm::get_targets` +### `peadm::get_targets` Type: Puppet Language @@ -756,7 +759,7 @@ Data type: `Optional[Integer[1,1]]` -### `peadm::node_manager_yaml_location` +### `peadm::node_manager_yaml_location` Type: Ruby 4.x API @@ -768,7 +771,7 @@ The peadm::node_manager_yaml_location function. Returns: `Any` -### `peadm::oid` +### `peadm::oid` Type: Puppet Language @@ -786,7 +789,7 @@ Data type: `String` -### `peadm::plan_step` +### `peadm::plan_step` Type: Ruby 4.x API @@ -810,7 +813,7 @@ Data type: `Callable` -### `peadm::recovery_opts_default` +### `peadm::recovery_opts_default` Type: Puppet Language @@ -822,7 +825,7 @@ The peadm::recovery_opts_default function. Returns: `Any` -### `peadm::update_pe_conf` +### `peadm::update_pe_conf` Type: Puppet Language @@ -846,7 +849,7 @@ Data type: `Hash` The hash to update the pe.conf file with -### `peadm::wait_until_service_ready` +### `peadm::wait_until_service_ready` Type: Puppet Language @@ -874,7 +877,7 @@ Data type: `TargetSpec` ## Data types -### `Peadm::Known_hosts` +### `Peadm::Known_hosts` The Peadm::Known_hosts data type. @@ -891,7 +894,7 @@ Array[Struct[ ]] ``` -### `Peadm::Ldap_config` +### `Peadm::Ldap_config` The Peadm::Ldap_config data type. @@ -925,19 +928,27 @@ Struct[{ }] ``` -### `Peadm::Pe_version` +### `Peadm::Pe_version` The Peadm::Pe_version data type. -Alias of `Pattern[/^\d+\.\d+\.\d+(-.+)?$/]` +Alias of + +```puppet +Pattern[/^\d+\.\d+\.\d+(-.+)?$/] +``` -### `Peadm::Pem` +### `Peadm::Pem` The Peadm::Pem data type. -Alias of `Pattern[/^-----BEGIN/]` +Alias of + +```puppet +Pattern[/^-----BEGIN/] +``` -### `Peadm::Recovery_opts` +### `Peadm::Recovery_opts` The Peadm::Recovery_opts data type. @@ -949,12 +960,12 @@ Struct[{ 'puppetdb' => Optional[Boolean], 'rbac' => Optional[Boolean], 'activity' => Optional[Boolean], - 'ca' => Optional[Boolean[false]], + 'ca' => Optional[Boolean], 'classifier' => Optional[Boolean], }] ``` -### `Peadm::SingleTargetSpec` +### `Peadm::SingleTargetSpec` A SingleTargetSpec represents any String, Target or single-element array of one or the other that can be passed to get_targets() to return an @@ -962,7 +973,11 @@ Array[Target, 1, 1]. This is a constrained type variant of Boltlib::TargetSpec for use when a _single_ target is valid, but multiple targets are not. -Alias of `Variant[Pattern[/\A[^[:space:],]+\z/], Target, Array[Peadm::SingleTargetSpec, 1, 1]]` +Alias of + +```puppet +Variant[Pattern[/\A[^[:space:],]+\z/], Target, Array[Peadm::SingleTargetSpec, 1, 1]] +``` ## Tasks @@ -1478,7 +1493,7 @@ Which port to query the status API on ## Plans -### `peadm::add_database` +### `peadm::add_database` The peadm::add_database class. @@ -1486,50 +1501,46 @@ The peadm::add_database class. The following parameters are available in the `peadm::add_database` plan: -* [`targets`](#-peadm--add_database--targets) -* [`primary_host`](#-peadm--add_database--primary_host) -* [`mode`](#-peadm--add_database--mode) -* [`begin_at_step`](#-peadm--add_database--begin_at_step) +* [`targets`](#targets) +* [`primary_host`](#primary_host) +* [`mode`](#mode) +* [`begin_at_step`](#begin_at_step) -##### `targets` +##### `targets` Data type: `Peadm::SingleTargetSpec` -##### `primary_host` +##### `primary_host` Data type: `Peadm::SingleTargetSpec` -##### `mode` +##### `mode` Data type: `Optional[Enum['init', 'pair']]` -Default value: `undef` +Default value: ``undef`` -##### `begin_at_step` +##### `begin_at_step` -Data type: - -```puppet -Optional[Enum[ +Data type: `Optional[Enum[ 'init-db-node', 'replicate-db', 'update-classification', 'update-db-settings', 'cleanup-db', - 'finalize']] -``` + 'finalize']]` -Default value: `undef` +Default value: ``undef`` -### `peadm::backup_ca` +### `peadm::backup_ca` The peadm::backup_ca class. @@ -1537,16 +1548,16 @@ The peadm::backup_ca class. The following parameters are available in the `peadm::backup_ca` plan: -* [`target`](#-peadm--backup_ca--target) -* [`output_directory`](#-peadm--backup_ca--output_directory) +* [`target`](#target) +* [`output_directory`](#output_directory) -##### `target` +##### `target` Data type: `Peadm::SingleTargetSpec` -##### `output_directory` +##### `output_directory` Data type: `Optional[String]` @@ -1554,7 +1565,7 @@ Data type: `Optional[String]` Default value: `'/tmp'` -### `peadm::convert` +### `peadm::convert` This plan sets required certificate extensions on PE nodes, and configures the required PE node groups to make an existing cluster compatible with @@ -1564,56 +1575,56 @@ management using PEAdm. The following parameters are available in the `peadm::convert` plan: -* [`primary_host`](#-peadm--convert--primary_host) -* [`replica_host`](#-peadm--convert--replica_host) -* [`compiler_hosts`](#-peadm--convert--compiler_hosts) -* [`primary_postgresql_host`](#-peadm--convert--primary_postgresql_host) -* [`replica_postgresql_host`](#-peadm--convert--replica_postgresql_host) -* [`compiler_pool_address`](#-peadm--convert--compiler_pool_address) -* [`internal_compiler_a_pool_address`](#-peadm--convert--internal_compiler_a_pool_address) -* [`internal_compiler_b_pool_address`](#-peadm--convert--internal_compiler_b_pool_address) -* [`dns_alt_names`](#-peadm--convert--dns_alt_names) -* [`begin_at_step`](#-peadm--convert--begin_at_step) +* [`primary_host`](#primary_host) +* [`replica_host`](#replica_host) +* [`compiler_hosts`](#compiler_hosts) +* [`primary_postgresql_host`](#primary_postgresql_host) +* [`replica_postgresql_host`](#replica_postgresql_host) +* [`compiler_pool_address`](#compiler_pool_address) +* [`internal_compiler_a_pool_address`](#internal_compiler_a_pool_address) +* [`internal_compiler_b_pool_address`](#internal_compiler_b_pool_address) +* [`dns_alt_names`](#dns_alt_names) +* [`begin_at_step`](#begin_at_step) -##### `primary_host` +##### `primary_host` Data type: `Peadm::SingleTargetSpec` -##### `replica_host` +##### `replica_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `compiler_hosts` +##### `compiler_hosts` Data type: `Optional[TargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `primary_postgresql_host` +##### `primary_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `replica_postgresql_host` +##### `replica_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `compiler_pool_address` +##### `compiler_pool_address` Data type: `String` @@ -1621,23 +1632,23 @@ Data type: `String` Default value: `$primary_host` -##### `internal_compiler_a_pool_address` +##### `internal_compiler_a_pool_address` Data type: `Optional[String]` -Default value: `undef` +Default value: ``undef`` -##### `internal_compiler_b_pool_address` +##### `internal_compiler_b_pool_address` Data type: `Optional[String]` -Default value: `undef` +Default value: ``undef`` -##### `dns_alt_names` +##### `dns_alt_names` Data type: `Array[String]` @@ -1645,23 +1656,19 @@ Data type: `Array[String]` Default value: `[]` -##### `begin_at_step` +##### `begin_at_step` -Data type: - -```puppet -Optional[Enum[ +Data type: `Optional[Enum[ 'modify-primary-certs', 'modify-infra-certs', 'convert-node-groups', - 'finalize']] -``` + 'finalize']]` -Default value: `undef` +Default value: ``undef`` -### `peadm::install` +### `peadm::install` Install a new PE cluster @@ -1669,45 +1676,45 @@ Install a new PE cluster The following parameters are available in the `peadm::install` plan: -* [`compiler_pool_address`](#-peadm--install--compiler_pool_address) -* [`internal_compiler_a_pool_address`](#-peadm--install--internal_compiler_a_pool_address) -* [`internal_compiler_b_pool_address`](#-peadm--install--internal_compiler_b_pool_address) -* [`pe_installer_source`](#-peadm--install--pe_installer_source) -* [`ldap_config`](#-peadm--install--ldap_config) -* [`final_agent_state`](#-peadm--install--final_agent_state) -* [`stagingdir`](#-peadm--install--stagingdir) -* [`uploaddir`](#-peadm--install--uploaddir) -* [`primary_host`](#-peadm--install--primary_host) -* [`replica_host`](#-peadm--install--replica_host) -* [`compiler_hosts`](#-peadm--install--compiler_hosts) -* [`primary_postgresql_host`](#-peadm--install--primary_postgresql_host) -* [`replica_postgresql_host`](#-peadm--install--replica_postgresql_host) -* [`console_password`](#-peadm--install--console_password) -* [`version`](#-peadm--install--version) -* [`dns_alt_names`](#-peadm--install--dns_alt_names) -* [`pe_conf_data`](#-peadm--install--pe_conf_data) -* [`code_manager_auto_configure`](#-peadm--install--code_manager_auto_configure) -* [`r10k_remote`](#-peadm--install--r10k_remote) -* [`r10k_private_key_file`](#-peadm--install--r10k_private_key_file) -* [`r10k_private_key_content`](#-peadm--install--r10k_private_key_content) -* [`r10k_known_hosts`](#-peadm--install--r10k_known_hosts) -* [`deploy_environment`](#-peadm--install--deploy_environment) -* [`license_key_file`](#-peadm--install--license_key_file) -* [`license_key_content`](#-peadm--install--license_key_content) -* [`download_mode`](#-peadm--install--download_mode) -* [`permit_unsafe_versions`](#-peadm--install--permit_unsafe_versions) -* [`token_lifetime`](#-peadm--install--token_lifetime) - -##### `compiler_pool_address` +* [`compiler_pool_address`](#compiler_pool_address) +* [`internal_compiler_a_pool_address`](#internal_compiler_a_pool_address) +* [`internal_compiler_b_pool_address`](#internal_compiler_b_pool_address) +* [`pe_installer_source`](#pe_installer_source) +* [`ldap_config`](#ldap_config) +* [`final_agent_state`](#final_agent_state) +* [`stagingdir`](#stagingdir) +* [`uploaddir`](#uploaddir) +* [`primary_host`](#primary_host) +* [`replica_host`](#replica_host) +* [`compiler_hosts`](#compiler_hosts) +* [`primary_postgresql_host`](#primary_postgresql_host) +* [`replica_postgresql_host`](#replica_postgresql_host) +* [`console_password`](#console_password) +* [`version`](#version) +* [`dns_alt_names`](#dns_alt_names) +* [`pe_conf_data`](#pe_conf_data) +* [`code_manager_auto_configure`](#code_manager_auto_configure) +* [`r10k_remote`](#r10k_remote) +* [`r10k_private_key_file`](#r10k_private_key_file) +* [`r10k_private_key_content`](#r10k_private_key_content) +* [`r10k_known_hosts`](#r10k_known_hosts) +* [`deploy_environment`](#deploy_environment) +* [`license_key_file`](#license_key_file) +* [`license_key_content`](#license_key_content) +* [`download_mode`](#download_mode) +* [`permit_unsafe_versions`](#permit_unsafe_versions) +* [`token_lifetime`](#token_lifetime) + +##### `compiler_pool_address` Data type: `Optional[String]` The service address used by agents to connect to compilers, or the Puppet service. Typically this is a load balancer. -Default value: `undef` +Default value: ``undef`` -##### `internal_compiler_a_pool_address` +##### `internal_compiler_a_pool_address` Data type: `Optional[String]` @@ -1715,9 +1722,9 @@ A load balancer address directing traffic to any of the "A" pool compilers. This is used for DR configuration in large and extra large architectures. -Default value: `undef` +Default value: ``undef`` -##### `internal_compiler_b_pool_address` +##### `internal_compiler_b_pool_address` Data type: `Optional[String]` @@ -1725,9 +1732,9 @@ A load balancer address directing traffic to any of the "B" pool compilers. This is used for DR configuration in large and extra large architectures. -Default value: `undef` +Default value: ``undef`` -##### `pe_installer_source` +##### `pe_installer_source` Data type: `Optional[String]` @@ -1736,9 +1743,9 @@ specified, PEAdm will attempt to download PE installation media from its standard public source. When specified, PEAdm will download directly from the URL given. -Default value: `undef` +Default value: ``undef`` -##### `ldap_config` +##### `ldap_config` Data type: `Optional[Peadm::Ldap_config]` @@ -1747,9 +1754,9 @@ The parameter should be set to a valid set of connection settings as documented for the PE RBAC /ds endpoint. See: https://puppet.com/docs/pe/latest/rbac_api_v1_directory.html#put_ds-request_format -Default value: `undef` +Default value: ``undef`` -##### `final_agent_state` +##### `final_agent_state` Data type: `Enum['running', 'stopped']` @@ -1758,69 +1765,69 @@ after PE is configured successfully. Default value: `'running'` -##### `stagingdir` +##### `stagingdir` Data type: `Optional[String]` Directory on the Bolt host where the installer tarball will be cached if download_mode is 'bolthost' (default) -Default value: `undef` +Default value: ``undef`` -##### `uploaddir` +##### `uploaddir` Data type: `Optional[String]` Directory the installer tarball will be uploaded to or expected to be in for offline usage. -Default value: `undef` +Default value: ``undef`` -##### `primary_host` +##### `primary_host` Data type: `Peadm::SingleTargetSpec` -##### `replica_host` +##### `replica_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `compiler_hosts` +##### `compiler_hosts` Data type: `Optional[TargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `primary_postgresql_host` +##### `primary_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `replica_postgresql_host` +##### `replica_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `console_password` +##### `console_password` Data type: `String` -##### `version` +##### `version` Data type: `Peadm::Pe_version` @@ -1828,15 +1835,15 @@ Data type: `Peadm::Pe_version` Default value: `'2021.7.4'` -##### `dns_alt_names` +##### `dns_alt_names` Data type: `Optional[Array[String]]` -Default value: `undef` +Default value: ``undef`` -##### `pe_conf_data` +##### `pe_conf_data` Data type: `Optional[Hash]` @@ -1844,71 +1851,71 @@ Data type: `Optional[Hash]` Default value: `{}` -##### `code_manager_auto_configure` +##### `code_manager_auto_configure` Data type: `Optional[Boolean]` -Default value: `undef` +Default value: ``undef`` -##### `r10k_remote` +##### `r10k_remote` Data type: `Optional[String]` -Default value: `undef` +Default value: ``undef`` -##### `r10k_private_key_file` +##### `r10k_private_key_file` Data type: `Optional[String]` -Default value: `undef` +Default value: ``undef`` -##### `r10k_private_key_content` +##### `r10k_private_key_content` Data type: `Optional[Peadm::Pem]` -Default value: `undef` +Default value: ``undef`` -##### `r10k_known_hosts` +##### `r10k_known_hosts` Data type: `Optional[Peadm::Known_hosts]` -Default value: `undef` +Default value: ``undef`` -##### `deploy_environment` +##### `deploy_environment` Data type: `Optional[String]` -Default value: `undef` +Default value: ``undef`` -##### `license_key_file` +##### `license_key_file` Data type: `Optional[String]` -Default value: `undef` +Default value: ``undef`` -##### `license_key_content` +##### `license_key_content` Data type: `Optional[String]` -Default value: `undef` +Default value: ``undef`` -##### `download_mode` +##### `download_mode` Data type: `Enum['direct', 'bolthost']` @@ -1916,15 +1923,15 @@ Data type: `Enum['direct', 'bolthost']` Default value: `'bolthost'` -##### `permit_unsafe_versions` +##### `permit_unsafe_versions` Data type: `Boolean` -Default value: `false` +Default value: ``false`` -##### `token_lifetime` +##### `token_lifetime` Data type: `String` @@ -1932,7 +1939,7 @@ Data type: `String` Default value: `'1y'` -### `peadm::modify_certificate` +### `peadm::modify_certificate` Certificates can be modified by adding extensions, removing extensions, or setting DNS alternative names. @@ -1941,26 +1948,26 @@ setting DNS alternative names. The following parameters are available in the `peadm::modify_certificate` plan: -* [`targets`](#-peadm--modify_certificate--targets) -* [`primary_host`](#-peadm--modify_certificate--primary_host) -* [`add_extensions`](#-peadm--modify_certificate--add_extensions) -* [`remove_extensions`](#-peadm--modify_certificate--remove_extensions) -* [`dns_alt_names`](#-peadm--modify_certificate--dns_alt_names) -* [`force_regenerate`](#-peadm--modify_certificate--force_regenerate) +* [`targets`](#targets) +* [`primary_host`](#primary_host) +* [`add_extensions`](#add_extensions) +* [`remove_extensions`](#remove_extensions) +* [`dns_alt_names`](#dns_alt_names) +* [`force_regenerate`](#force_regenerate) -##### `targets` +##### `targets` Data type: `TargetSpec` -##### `primary_host` +##### `primary_host` Data type: `Peadm::SingleTargetSpec` -##### `add_extensions` +##### `add_extensions` Data type: `Hash` @@ -1968,7 +1975,7 @@ Data type: `Hash` Default value: `{}` -##### `remove_extensions` +##### `remove_extensions` Data type: `Array` @@ -1976,23 +1983,23 @@ Data type: `Array` Default value: `[]` -##### `dns_alt_names` +##### `dns_alt_names` Data type: `Optional[Array]` -Default value: `undef` +Default value: ``undef`` -##### `force_regenerate` +##### `force_regenerate` Data type: `Boolean` -Default value: `false` +Default value: ``false`` -### `peadm::restore_ca` +### `peadm::restore_ca` The peadm::restore_ca class. @@ -2000,23 +2007,23 @@ The peadm::restore_ca class. The following parameters are available in the `peadm::restore_ca` plan: -* [`target`](#-peadm--restore_ca--target) -* [`file_path`](#-peadm--restore_ca--file_path) -* [`recovery_directory`](#-peadm--restore_ca--recovery_directory) +* [`target`](#target) +* [`file_path`](#file_path) +* [`recovery_directory`](#recovery_directory) -##### `target` +##### `target` Data type: `Peadm::SingleTargetSpec` -##### `file_path` +##### `file_path` Data type: `String` -##### `recovery_directory` +##### `recovery_directory` Data type: `Optional[String]` @@ -2024,7 +2031,7 @@ Data type: `Optional[String]` Default value: `'/tmp/peadm_recovery'` -### `peadm::status` +### `peadm::status` Return status information from one or more PE clusters in a table format @@ -2040,19 +2047,19 @@ peadm::status($targets, 'table', true, true) The following parameters are available in the `peadm::status` plan: -* [`targets`](#-peadm--status--targets) -* [`format`](#-peadm--status--format) -* [`summarize`](#-peadm--status--summarize) -* [`verbose`](#-peadm--status--verbose) -* [`colors`](#-peadm--status--colors) +* [`targets`](#targets) +* [`format`](#format) +* [`summarize`](#summarize) +* [`verbose`](#verbose) +* [`colors`](#colors) -##### `targets` +##### `targets` Data type: `TargetSpec` These are a list of the primary puppetservers from one or multiple puppet stacks -##### `format` +##### `format` Data type: `Enum[json,table]` @@ -2060,31 +2067,31 @@ The output format to dump to stdout (json or table) Default value: `'table'` -##### `summarize` +##### `summarize` Data type: `Boolean` Controls the type of json output to render, defaults to true -Default value: `true` +Default value: ``true`` -##### `verbose` +##### `verbose` Data type: `Boolean` Toggles the output to show all the operationally services, can be loads more data -Default value: `false` +Default value: ``false`` -##### `colors` +##### `colors` Data type: `Boolean` Toggles the usage of colors, you may want to disable if the format is json -Default value: `$format ? { 'json' => false, default => true` +Default value: `$format` -### `peadm::upgrade` +### `peadm::upgrade` Upgrade a PEAdm-managed cluster @@ -2092,35 +2099,35 @@ Upgrade a PEAdm-managed cluster The following parameters are available in the `peadm::upgrade` plan: -* [`compiler_pool_address`](#-peadm--upgrade--compiler_pool_address) -* [`internal_compiler_a_pool_address`](#-peadm--upgrade--internal_compiler_a_pool_address) -* [`internal_compiler_b_pool_address`](#-peadm--upgrade--internal_compiler_b_pool_address) -* [`pe_installer_source`](#-peadm--upgrade--pe_installer_source) -* [`final_agent_state`](#-peadm--upgrade--final_agent_state) -* [`r10k_known_hosts`](#-peadm--upgrade--r10k_known_hosts) -* [`stagingdir`](#-peadm--upgrade--stagingdir) -* [`uploaddir`](#-peadm--upgrade--uploaddir) -* [`primary_host`](#-peadm--upgrade--primary_host) -* [`replica_host`](#-peadm--upgrade--replica_host) -* [`compiler_hosts`](#-peadm--upgrade--compiler_hosts) -* [`primary_postgresql_host`](#-peadm--upgrade--primary_postgresql_host) -* [`replica_postgresql_host`](#-peadm--upgrade--replica_postgresql_host) -* [`version`](#-peadm--upgrade--version) -* [`token_file`](#-peadm--upgrade--token_file) -* [`download_mode`](#-peadm--upgrade--download_mode) -* [`permit_unsafe_versions`](#-peadm--upgrade--permit_unsafe_versions) -* [`begin_at_step`](#-peadm--upgrade--begin_at_step) - -##### `compiler_pool_address` +* [`compiler_pool_address`](#compiler_pool_address) +* [`internal_compiler_a_pool_address`](#internal_compiler_a_pool_address) +* [`internal_compiler_b_pool_address`](#internal_compiler_b_pool_address) +* [`pe_installer_source`](#pe_installer_source) +* [`final_agent_state`](#final_agent_state) +* [`r10k_known_hosts`](#r10k_known_hosts) +* [`stagingdir`](#stagingdir) +* [`uploaddir`](#uploaddir) +* [`primary_host`](#primary_host) +* [`replica_host`](#replica_host) +* [`compiler_hosts`](#compiler_hosts) +* [`primary_postgresql_host`](#primary_postgresql_host) +* [`replica_postgresql_host`](#replica_postgresql_host) +* [`version`](#version) +* [`token_file`](#token_file) +* [`download_mode`](#download_mode) +* [`permit_unsafe_versions`](#permit_unsafe_versions) +* [`begin_at_step`](#begin_at_step) + +##### `compiler_pool_address` Data type: `Optional[String]` The service address used by agents to connect to compilers, or the Puppet service. Typically this is a load balancer. -Default value: `undef` +Default value: ``undef`` -##### `internal_compiler_a_pool_address` +##### `internal_compiler_a_pool_address` Data type: `Optional[String]` @@ -2128,9 +2135,9 @@ A load balancer address directing traffic to any of the "A" pool compilers. This is used for DR configuration in large and extra large architectures. -Default value: `undef` +Default value: ``undef`` -##### `internal_compiler_b_pool_address` +##### `internal_compiler_b_pool_address` Data type: `Optional[String]` @@ -2138,9 +2145,9 @@ A load balancer address directing traffic to any of the "B" pool compilers. This is used for DR configuration in large and extra large architectures. -Default value: `undef` +Default value: ``undef`` -##### `pe_installer_source` +##### `pe_installer_source` Data type: `Optional[String]` @@ -2149,9 +2156,9 @@ specified, PEAdm will attempt to download PE installation media from its standard public source. When specified, PEAdm will download directly from the URL given. -Default value: `undef` +Default value: ``undef`` -##### `final_agent_state` +##### `final_agent_state` Data type: `Enum['running', 'stopped']` @@ -2160,7 +2167,7 @@ after PE is upgraded successfully. Default value: `'running'` -##### `r10k_known_hosts` +##### `r10k_known_hosts` Data type: `Optional[Peadm::Known_hosts]` @@ -2169,9 +2176,9 @@ r10k_remote host when using ssh. you must provide \$r10k_known_hosts information in the form of an array of hashes with 'name', 'type' and 'key' information for hostname, key-type and public key. -Default value: `undef` +Default value: ``undef`` -##### `stagingdir` +##### `stagingdir` Data type: `String` @@ -2180,7 +2187,7 @@ download_mode is 'bolthost' (default) Default value: `'/tmp'` -##### `uploaddir` +##### `uploaddir` Data type: `String` @@ -2189,61 +2196,61 @@ for offline usage. Default value: `'/tmp'` -##### `primary_host` +##### `primary_host` Data type: `Peadm::SingleTargetSpec` -##### `replica_host` +##### `replica_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `compiler_hosts` +##### `compiler_hosts` Data type: `Optional[TargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `primary_postgresql_host` +##### `primary_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `replica_postgresql_host` +##### `replica_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: `undef` +Default value: ``undef`` -##### `version` +##### `version` Data type: `Optional[Peadm::Pe_version]` -Default value: `undef` +Default value: ``undef`` -##### `token_file` +##### `token_file` Data type: `Optional[String]` -Default value: `undef` +Default value: ``undef`` -##### `download_mode` +##### `download_mode` Data type: `Enum[direct,bolthost]` @@ -2251,29 +2258,25 @@ Data type: `Enum[direct,bolthost]` Default value: `'bolthost'` -##### `permit_unsafe_versions` +##### `permit_unsafe_versions` Data type: `Boolean` -Default value: `false` +Default value: ``false`` -##### `begin_at_step` +##### `begin_at_step` -Data type: - -```puppet -Optional[Enum[ +Data type: `Optional[Enum[ 'upgrade-primary', 'upgrade-node-groups', 'upgrade-primary-compilers', 'upgrade-replica', 'upgrade-replica-compilers', - 'finalize']] -``` + 'finalize']]` -Default value: `undef` +Default value: ``undef`` From c7151eec9fe4e4c9bb2720328869a76ad50bd549 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 21 Nov 2023 13:45:59 +0100 Subject: [PATCH 025/110] use cert auth for pdb import --- plans/restore.pp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/plans/restore.pp b/plans/restore.pp index 8b53277c..6aa74c81 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -222,7 +222,10 @@ # or other factors. if getvar('recovery_opts.puppetdb') { run_command(@("CMD"/L), $primary_target) - /opt/puppetlabs/bin/puppet-db import ${shellquote($recovery_directory)}/puppetdb-archive.bin + /opt/puppetlabs/bin/puppet-db import \ + --cert=$(/opt/puppetlabs/bin/puppet config print hostcert) \ + --key=$(/opt/puppetlabs/bin/puppet config print hostprivkey) \ + ${shellquote($recovery_directory)}/puppetdb-archive.bin | CMD # lint:endignore } From dfbf1f840a5154fb64a180d9e5f934f24601310a Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 21 Nov 2023 13:49:09 +0100 Subject: [PATCH 026/110] regenerate REFERENCE.md using the rake task --- REFERENCE.md | 603 +++++++++++++++++++++++++-------------------------- 1 file changed, 301 insertions(+), 302 deletions(-) diff --git a/REFERENCE.md b/REFERENCE.md index 1f6332d1..72791aaf 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -6,9 +6,6 @@ ### Classes -#### Public Classes - - #### Private Classes * `peadm::setup::convert_node_manager`: Used during the peadm::convert plan @@ -18,37 +15,37 @@ ### Functions -* [`peadm::assert_supported_architecture`](#peadmassert_supported_architecture): Assert that the architecture given is a supported one -* [`peadm::assert_supported_bolt_version`](#peadmassert_supported_bolt_version): Assert that the Bolt executable running PEAdm is a supported version -* [`peadm::assert_supported_pe_version`](#peadmassert_supported_pe_version): Assert that the PE version given is supported by PEAdm -* [`peadm::bolt_version`](#peadmbolt_version) -* [`peadm::certname`](#peadmcertname): Return the certname of the given target-like input -* [`peadm::check_version_and_known_hosts`](#peadmcheck_version_and_known_hosts): Checks PE verison and warns about setting r10k_known_hosts -* [`peadm::convert_hash`](#peadmconvert_hash): converts two arrays into hash -* [`peadm::convert_status`](#peadmconvert_status): Transforms a value in a human readable status with or without colors -* [`peadm::determine_status`](#peadmdetermine_status): Produces a summarized hash of the given status data -* [`peadm::fail_on_transport`](#peadmfail_on_transport): Fails if any nodes have the chosen transport. Useful for excluding PCP when it's not appopriate -* [`peadm::file_content_upload`](#peadmfile_content_upload) -* [`peadm::file_or_content`](#peadmfile_or_content) -* [`peadm::flatten_compact`](#peadmflatten_compact) -* [`peadm::generate_pe_conf`](#peadmgenerate_pe_conf): Generate a pe.conf file in JSON format -* [`peadm::get_pe_conf`](#peadmget_pe_conf) -* [`peadm::get_targets`](#peadmget_targets): Accept undef or a SingleTargetSpec, and return an Array[Target, 1, 0]. This differs from get_target() in that: - It returns an Array[Target -* [`peadm::node_manager_yaml_location`](#peadmnode_manager_yaml_location) -* [`peadm::oid`](#peadmoid) -* [`peadm::plan_step`](#peadmplan_step) -* [`peadm::recovery_opts_default`](#peadmrecovery_opts_default) -* [`peadm::update_pe_conf`](#peadmupdate_pe_conf): Update the pe.conf file on a target with the provided hash -* [`peadm::wait_until_service_ready`](#peadmwait_until_service_ready): A convenience function to help remember port numbers for services and handle running the wait_until_service_ready task +* [`peadm::assert_supported_architecture`](#peadm--assert_supported_architecture): Assert that the architecture given is a supported one +* [`peadm::assert_supported_bolt_version`](#peadm--assert_supported_bolt_version): Assert that the Bolt executable running PEAdm is a supported version +* [`peadm::assert_supported_pe_version`](#peadm--assert_supported_pe_version): Assert that the PE version given is supported by PEAdm +* [`peadm::bolt_version`](#peadm--bolt_version) +* [`peadm::certname`](#peadm--certname): Return the certname of the given target-like input +* [`peadm::check_version_and_known_hosts`](#peadm--check_version_and_known_hosts): Checks PE verison and warns about setting r10k_known_hosts +* [`peadm::convert_hash`](#peadm--convert_hash): converts two arrays into hash +* [`peadm::convert_status`](#peadm--convert_status): Transforms a value in a human readable status with or without colors +* [`peadm::determine_status`](#peadm--determine_status): Produces a summarized hash of the given status data +* [`peadm::fail_on_transport`](#peadm--fail_on_transport): Fails if any nodes have the chosen transport. Useful for excluding PCP when it's not appopriate +* [`peadm::file_content_upload`](#peadm--file_content_upload) +* [`peadm::file_or_content`](#peadm--file_or_content) +* [`peadm::flatten_compact`](#peadm--flatten_compact) +* [`peadm::generate_pe_conf`](#peadm--generate_pe_conf): Generate a pe.conf file in JSON format +* [`peadm::get_pe_conf`](#peadm--get_pe_conf) +* [`peadm::get_targets`](#peadm--get_targets): Accept undef or a SingleTargetSpec, and return an Array[Target, 1, 0]. This differs from get_target() in that: - It returns an Array[Target +* [`peadm::node_manager_yaml_location`](#peadm--node_manager_yaml_location) +* [`peadm::oid`](#peadm--oid) +* [`peadm::plan_step`](#peadm--plan_step) +* [`peadm::recovery_opts_default`](#peadm--recovery_opts_default) +* [`peadm::update_pe_conf`](#peadm--update_pe_conf): Update the pe.conf file on a target with the provided hash +* [`peadm::wait_until_service_ready`](#peadm--wait_until_service_ready): A convenience function to help remember port numbers for services and handle running the wait_until_service_ready task ### Data types -* [`Peadm::Known_hosts`](#peadmknown_hosts) -* [`Peadm::Ldap_config`](#peadmldap_config) -* [`Peadm::Pe_version`](#peadmpe_version) -* [`Peadm::Pem`](#peadmpem) -* [`Peadm::Recovery_opts`](#peadmrecovery_opts) -* [`Peadm::SingleTargetSpec`](#peadmsingletargetspec): A SingleTargetSpec represents any String, Target or single-element array of one or the other that can be passed to get_targets() to return an +* [`Peadm::Known_hosts`](#Peadm--Known_hosts) +* [`Peadm::Ldap_config`](#Peadm--Ldap_config) +* [`Peadm::Pe_version`](#Peadm--Pe_version) +* [`Peadm::Pem`](#Peadm--Pem) +* [`Peadm::Recovery_opts`](#Peadm--Recovery_opts) +* [`Peadm::SingleTargetSpec`](#Peadm--SingleTargetSpec): A SingleTargetSpec represents any String, Target or single-element array of one or the other that can be passed to get_targets() to return an ### Tasks @@ -87,14 +84,14 @@ #### Public Plans -* [`peadm::add_database`](#peadmadd_database) -* [`peadm::backup_ca`](#peadmbackup_ca) -* [`peadm::convert`](#peadmconvert): Convert an existing PE cluster to a PEAdm-managed cluster -* [`peadm::install`](#peadminstall): Install a new PE cluster -* [`peadm::modify_certificate`](#peadmmodify_certificate): Modify the certificate of one or more targets -* [`peadm::restore_ca`](#peadmrestore_ca) -* [`peadm::status`](#peadmstatus): Return status information from one or more PE clusters in a table format -* [`peadm::upgrade`](#peadmupgrade): Upgrade a PEAdm-managed cluster +* [`peadm::add_database`](#peadm--add_database) +* [`peadm::backup_ca`](#peadm--backup_ca) +* [`peadm::convert`](#peadm--convert): Convert an existing PE cluster to a PEAdm-managed cluster +* [`peadm::install`](#peadm--install): Install a new PE cluster +* [`peadm::modify_certificate`](#peadm--modify_certificate): Modify the certificate of one or more targets +* [`peadm::restore_ca`](#peadm--restore_ca) +* [`peadm::status`](#peadm--status): Return status information from one or more PE clusters in a table format +* [`peadm::upgrade`](#peadm--upgrade): Upgrade a PEAdm-managed cluster #### Private Plans @@ -123,11 +120,9 @@ Supported use cases: * `peadm::util::update_classification`: Configure classification * `peadm::util::update_db_setting`: Make updates to PuppetDB database settings -## Classes - ## Functions -### `peadm::assert_supported_architecture` +### `peadm::assert_supported_architecture` Type: Puppet Language @@ -169,7 +164,7 @@ Data type: `Variant[TargetSpec, Undef]` -### `peadm::assert_supported_bolt_version` +### `peadm::assert_supported_bolt_version` Type: Puppet Language @@ -185,7 +180,7 @@ Accepts a parameter for the $supported_bolt_version for unit testing purposes Returns: `Struct[{ 'supported' => Boolean }]` -### `peadm::assert_supported_pe_version` +### `peadm::assert_supported_pe_version` Type: Puppet Language @@ -215,7 +210,7 @@ Data type: `Boolean` -### `peadm::bolt_version` +### `peadm::bolt_version` Type: Ruby 4.x API @@ -227,7 +222,7 @@ The peadm::bolt_version function. Returns: `Any` -### `peadm::certname` +### `peadm::certname` Type: Puppet Language @@ -259,17 +254,21 @@ Returns: `Variant[String, Undef]` ##### `target` -Data type: `Variant[Target, +Data type: + +```puppet +Variant[Target, String, Undef, Array[Target,1,1], Array[String,1,1], Array[Undef,1,1], - Array[Any,0,0]]` + Array[Any,0,0]] +``` -### `peadm::check_version_and_known_hosts` +### `peadm::check_version_and_known_hosts` Type: Puppet Language @@ -319,7 +318,7 @@ Data type: `Optional[Peadm::Known_hosts]` -### `peadm::convert_hash` +### `peadm::convert_hash` Type: Puppet Language @@ -365,7 +364,7 @@ Data type: `Array[Array]` data to be merged into an array with the keys -### `peadm::convert_status` +### `peadm::convert_status` Type: Puppet Language @@ -460,7 +459,7 @@ Data type: `Optional[Boolean]` Adds colors to the status, defaults to true -### `peadm::determine_status` +### `peadm::determine_status` Type: Puppet Language @@ -580,7 +579,7 @@ Data type: `Boolean` Adds colors to the status, defaults to true -### `peadm::fail_on_transport` +### `peadm::fail_on_transport` Type: Puppet Language @@ -614,7 +613,7 @@ Data type: `String` -### `peadm::file_content_upload` +### `peadm::file_content_upload` Type: Ruby 4.x API @@ -644,7 +643,7 @@ Data type: `TargetOrTargets` -### `peadm::file_or_content` +### `peadm::file_or_content` Type: Puppet Language @@ -674,7 +673,7 @@ Data type: `Variant[String, Undef]` -### `peadm::flatten_compact` +### `peadm::flatten_compact` Type: Puppet Language @@ -692,7 +691,7 @@ Data type: `Array` -### `peadm::generate_pe_conf` +### `peadm::generate_pe_conf` Type: Puppet Language @@ -711,7 +710,7 @@ Data type: `Hash` A hash of settings to set in the config file. Any keys that are set to undef will not be included in the config file. -### `peadm::get_pe_conf` +### `peadm::get_pe_conf` Type: Puppet Language @@ -729,7 +728,7 @@ Data type: `Target` -### `peadm::get_targets` +### `peadm::get_targets` Type: Puppet Language @@ -759,7 +758,7 @@ Data type: `Optional[Integer[1,1]]` -### `peadm::node_manager_yaml_location` +### `peadm::node_manager_yaml_location` Type: Ruby 4.x API @@ -771,7 +770,7 @@ The peadm::node_manager_yaml_location function. Returns: `Any` -### `peadm::oid` +### `peadm::oid` Type: Puppet Language @@ -789,7 +788,7 @@ Data type: `String` -### `peadm::plan_step` +### `peadm::plan_step` Type: Ruby 4.x API @@ -813,7 +812,7 @@ Data type: `Callable` -### `peadm::recovery_opts_default` +### `peadm::recovery_opts_default` Type: Puppet Language @@ -825,7 +824,7 @@ The peadm::recovery_opts_default function. Returns: `Any` -### `peadm::update_pe_conf` +### `peadm::update_pe_conf` Type: Puppet Language @@ -849,7 +848,7 @@ Data type: `Hash` The hash to update the pe.conf file with -### `peadm::wait_until_service_ready` +### `peadm::wait_until_service_ready` Type: Puppet Language @@ -877,7 +876,7 @@ Data type: `TargetSpec` ## Data types -### `Peadm::Known_hosts` +### `Peadm::Known_hosts` The Peadm::Known_hosts data type. @@ -894,7 +893,7 @@ Array[Struct[ ]] ``` -### `Peadm::Ldap_config` +### `Peadm::Ldap_config` The Peadm::Ldap_config data type. @@ -928,27 +927,19 @@ Struct[{ }] ``` -### `Peadm::Pe_version` +### `Peadm::Pe_version` The Peadm::Pe_version data type. -Alias of - -```puppet -Pattern[/^\d+\.\d+\.\d+(-.+)?$/] -``` +Alias of `Pattern[/^\d+\.\d+\.\d+(-.+)?$/]` -### `Peadm::Pem` +### `Peadm::Pem` The Peadm::Pem data type. -Alias of - -```puppet -Pattern[/^-----BEGIN/] -``` +Alias of `Pattern[/^-----BEGIN/]` -### `Peadm::Recovery_opts` +### `Peadm::Recovery_opts` The Peadm::Recovery_opts data type. @@ -965,7 +956,7 @@ Struct[{ }] ``` -### `Peadm::SingleTargetSpec` +### `Peadm::SingleTargetSpec` A SingleTargetSpec represents any String, Target or single-element array of one or the other that can be passed to get_targets() to return an @@ -973,11 +964,7 @@ Array[Target, 1, 1]. This is a constrained type variant of Boltlib::TargetSpec for use when a _single_ target is valid, but multiple targets are not. -Alias of - -```puppet -Variant[Pattern[/\A[^[:space:],]+\z/], Target, Array[Peadm::SingleTargetSpec, 1, 1]] -``` +Alias of `Variant[Pattern[/\A[^[:space:],]+\z/], Target, Array[Peadm::SingleTargetSpec, 1, 1]]` ## Tasks @@ -1493,7 +1480,7 @@ Which port to query the status API on ## Plans -### `peadm::add_database` +### `peadm::add_database` The peadm::add_database class. @@ -1501,46 +1488,50 @@ The peadm::add_database class. The following parameters are available in the `peadm::add_database` plan: -* [`targets`](#targets) -* [`primary_host`](#primary_host) -* [`mode`](#mode) -* [`begin_at_step`](#begin_at_step) +* [`targets`](#-peadm--add_database--targets) +* [`primary_host`](#-peadm--add_database--primary_host) +* [`mode`](#-peadm--add_database--mode) +* [`begin_at_step`](#-peadm--add_database--begin_at_step) -##### `targets` +##### `targets` Data type: `Peadm::SingleTargetSpec` -##### `primary_host` +##### `primary_host` Data type: `Peadm::SingleTargetSpec` -##### `mode` +##### `mode` Data type: `Optional[Enum['init', 'pair']]` -Default value: ``undef`` +Default value: `undef` -##### `begin_at_step` +##### `begin_at_step` -Data type: `Optional[Enum[ +Data type: + +```puppet +Optional[Enum[ 'init-db-node', 'replicate-db', 'update-classification', 'update-db-settings', 'cleanup-db', - 'finalize']]` + 'finalize']] +``` -Default value: ``undef`` +Default value: `undef` -### `peadm::backup_ca` +### `peadm::backup_ca` The peadm::backup_ca class. @@ -1548,16 +1539,16 @@ The peadm::backup_ca class. The following parameters are available in the `peadm::backup_ca` plan: -* [`target`](#target) -* [`output_directory`](#output_directory) +* [`target`](#-peadm--backup_ca--target) +* [`output_directory`](#-peadm--backup_ca--output_directory) -##### `target` +##### `target` Data type: `Peadm::SingleTargetSpec` -##### `output_directory` +##### `output_directory` Data type: `Optional[String]` @@ -1565,7 +1556,7 @@ Data type: `Optional[String]` Default value: `'/tmp'` -### `peadm::convert` +### `peadm::convert` This plan sets required certificate extensions on PE nodes, and configures the required PE node groups to make an existing cluster compatible with @@ -1575,56 +1566,56 @@ management using PEAdm. The following parameters are available in the `peadm::convert` plan: -* [`primary_host`](#primary_host) -* [`replica_host`](#replica_host) -* [`compiler_hosts`](#compiler_hosts) -* [`primary_postgresql_host`](#primary_postgresql_host) -* [`replica_postgresql_host`](#replica_postgresql_host) -* [`compiler_pool_address`](#compiler_pool_address) -* [`internal_compiler_a_pool_address`](#internal_compiler_a_pool_address) -* [`internal_compiler_b_pool_address`](#internal_compiler_b_pool_address) -* [`dns_alt_names`](#dns_alt_names) -* [`begin_at_step`](#begin_at_step) +* [`primary_host`](#-peadm--convert--primary_host) +* [`replica_host`](#-peadm--convert--replica_host) +* [`compiler_hosts`](#-peadm--convert--compiler_hosts) +* [`primary_postgresql_host`](#-peadm--convert--primary_postgresql_host) +* [`replica_postgresql_host`](#-peadm--convert--replica_postgresql_host) +* [`compiler_pool_address`](#-peadm--convert--compiler_pool_address) +* [`internal_compiler_a_pool_address`](#-peadm--convert--internal_compiler_a_pool_address) +* [`internal_compiler_b_pool_address`](#-peadm--convert--internal_compiler_b_pool_address) +* [`dns_alt_names`](#-peadm--convert--dns_alt_names) +* [`begin_at_step`](#-peadm--convert--begin_at_step) -##### `primary_host` +##### `primary_host` Data type: `Peadm::SingleTargetSpec` -##### `replica_host` +##### `replica_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `compiler_hosts` +##### `compiler_hosts` Data type: `Optional[TargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `primary_postgresql_host` +##### `primary_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `replica_postgresql_host` +##### `replica_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `compiler_pool_address` +##### `compiler_pool_address` Data type: `String` @@ -1632,23 +1623,23 @@ Data type: `String` Default value: `$primary_host` -##### `internal_compiler_a_pool_address` +##### `internal_compiler_a_pool_address` Data type: `Optional[String]` -Default value: ``undef`` +Default value: `undef` -##### `internal_compiler_b_pool_address` +##### `internal_compiler_b_pool_address` Data type: `Optional[String]` -Default value: ``undef`` +Default value: `undef` -##### `dns_alt_names` +##### `dns_alt_names` Data type: `Array[String]` @@ -1656,19 +1647,23 @@ Data type: `Array[String]` Default value: `[]` -##### `begin_at_step` +##### `begin_at_step` -Data type: `Optional[Enum[ +Data type: + +```puppet +Optional[Enum[ 'modify-primary-certs', 'modify-infra-certs', 'convert-node-groups', - 'finalize']]` + 'finalize']] +``` -Default value: ``undef`` +Default value: `undef` -### `peadm::install` +### `peadm::install` Install a new PE cluster @@ -1676,45 +1671,45 @@ Install a new PE cluster The following parameters are available in the `peadm::install` plan: -* [`compiler_pool_address`](#compiler_pool_address) -* [`internal_compiler_a_pool_address`](#internal_compiler_a_pool_address) -* [`internal_compiler_b_pool_address`](#internal_compiler_b_pool_address) -* [`pe_installer_source`](#pe_installer_source) -* [`ldap_config`](#ldap_config) -* [`final_agent_state`](#final_agent_state) -* [`stagingdir`](#stagingdir) -* [`uploaddir`](#uploaddir) -* [`primary_host`](#primary_host) -* [`replica_host`](#replica_host) -* [`compiler_hosts`](#compiler_hosts) -* [`primary_postgresql_host`](#primary_postgresql_host) -* [`replica_postgresql_host`](#replica_postgresql_host) -* [`console_password`](#console_password) -* [`version`](#version) -* [`dns_alt_names`](#dns_alt_names) -* [`pe_conf_data`](#pe_conf_data) -* [`code_manager_auto_configure`](#code_manager_auto_configure) -* [`r10k_remote`](#r10k_remote) -* [`r10k_private_key_file`](#r10k_private_key_file) -* [`r10k_private_key_content`](#r10k_private_key_content) -* [`r10k_known_hosts`](#r10k_known_hosts) -* [`deploy_environment`](#deploy_environment) -* [`license_key_file`](#license_key_file) -* [`license_key_content`](#license_key_content) -* [`download_mode`](#download_mode) -* [`permit_unsafe_versions`](#permit_unsafe_versions) -* [`token_lifetime`](#token_lifetime) - -##### `compiler_pool_address` +* [`compiler_pool_address`](#-peadm--install--compiler_pool_address) +* [`internal_compiler_a_pool_address`](#-peadm--install--internal_compiler_a_pool_address) +* [`internal_compiler_b_pool_address`](#-peadm--install--internal_compiler_b_pool_address) +* [`pe_installer_source`](#-peadm--install--pe_installer_source) +* [`ldap_config`](#-peadm--install--ldap_config) +* [`final_agent_state`](#-peadm--install--final_agent_state) +* [`stagingdir`](#-peadm--install--stagingdir) +* [`uploaddir`](#-peadm--install--uploaddir) +* [`primary_host`](#-peadm--install--primary_host) +* [`replica_host`](#-peadm--install--replica_host) +* [`compiler_hosts`](#-peadm--install--compiler_hosts) +* [`primary_postgresql_host`](#-peadm--install--primary_postgresql_host) +* [`replica_postgresql_host`](#-peadm--install--replica_postgresql_host) +* [`console_password`](#-peadm--install--console_password) +* [`version`](#-peadm--install--version) +* [`dns_alt_names`](#-peadm--install--dns_alt_names) +* [`pe_conf_data`](#-peadm--install--pe_conf_data) +* [`code_manager_auto_configure`](#-peadm--install--code_manager_auto_configure) +* [`r10k_remote`](#-peadm--install--r10k_remote) +* [`r10k_private_key_file`](#-peadm--install--r10k_private_key_file) +* [`r10k_private_key_content`](#-peadm--install--r10k_private_key_content) +* [`r10k_known_hosts`](#-peadm--install--r10k_known_hosts) +* [`deploy_environment`](#-peadm--install--deploy_environment) +* [`license_key_file`](#-peadm--install--license_key_file) +* [`license_key_content`](#-peadm--install--license_key_content) +* [`download_mode`](#-peadm--install--download_mode) +* [`permit_unsafe_versions`](#-peadm--install--permit_unsafe_versions) +* [`token_lifetime`](#-peadm--install--token_lifetime) + +##### `compiler_pool_address` Data type: `Optional[String]` The service address used by agents to connect to compilers, or the Puppet service. Typically this is a load balancer. -Default value: ``undef`` +Default value: `undef` -##### `internal_compiler_a_pool_address` +##### `internal_compiler_a_pool_address` Data type: `Optional[String]` @@ -1722,9 +1717,9 @@ A load balancer address directing traffic to any of the "A" pool compilers. This is used for DR configuration in large and extra large architectures. -Default value: ``undef`` +Default value: `undef` -##### `internal_compiler_b_pool_address` +##### `internal_compiler_b_pool_address` Data type: `Optional[String]` @@ -1732,9 +1727,9 @@ A load balancer address directing traffic to any of the "B" pool compilers. This is used for DR configuration in large and extra large architectures. -Default value: ``undef`` +Default value: `undef` -##### `pe_installer_source` +##### `pe_installer_source` Data type: `Optional[String]` @@ -1743,9 +1738,9 @@ specified, PEAdm will attempt to download PE installation media from its standard public source. When specified, PEAdm will download directly from the URL given. -Default value: ``undef`` +Default value: `undef` -##### `ldap_config` +##### `ldap_config` Data type: `Optional[Peadm::Ldap_config]` @@ -1754,9 +1749,9 @@ The parameter should be set to a valid set of connection settings as documented for the PE RBAC /ds endpoint. See: https://puppet.com/docs/pe/latest/rbac_api_v1_directory.html#put_ds-request_format -Default value: ``undef`` +Default value: `undef` -##### `final_agent_state` +##### `final_agent_state` Data type: `Enum['running', 'stopped']` @@ -1765,69 +1760,69 @@ after PE is configured successfully. Default value: `'running'` -##### `stagingdir` +##### `stagingdir` Data type: `Optional[String]` Directory on the Bolt host where the installer tarball will be cached if download_mode is 'bolthost' (default) -Default value: ``undef`` +Default value: `undef` -##### `uploaddir` +##### `uploaddir` Data type: `Optional[String]` Directory the installer tarball will be uploaded to or expected to be in for offline usage. -Default value: ``undef`` +Default value: `undef` -##### `primary_host` +##### `primary_host` Data type: `Peadm::SingleTargetSpec` -##### `replica_host` +##### `replica_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `compiler_hosts` +##### `compiler_hosts` Data type: `Optional[TargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `primary_postgresql_host` +##### `primary_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `replica_postgresql_host` +##### `replica_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `console_password` +##### `console_password` Data type: `String` -##### `version` +##### `version` Data type: `Peadm::Pe_version` @@ -1835,15 +1830,15 @@ Data type: `Peadm::Pe_version` Default value: `'2021.7.4'` -##### `dns_alt_names` +##### `dns_alt_names` Data type: `Optional[Array[String]]` -Default value: ``undef`` +Default value: `undef` -##### `pe_conf_data` +##### `pe_conf_data` Data type: `Optional[Hash]` @@ -1851,71 +1846,71 @@ Data type: `Optional[Hash]` Default value: `{}` -##### `code_manager_auto_configure` +##### `code_manager_auto_configure` Data type: `Optional[Boolean]` -Default value: ``undef`` +Default value: `undef` -##### `r10k_remote` +##### `r10k_remote` Data type: `Optional[String]` -Default value: ``undef`` +Default value: `undef` -##### `r10k_private_key_file` +##### `r10k_private_key_file` Data type: `Optional[String]` -Default value: ``undef`` +Default value: `undef` -##### `r10k_private_key_content` +##### `r10k_private_key_content` Data type: `Optional[Peadm::Pem]` -Default value: ``undef`` +Default value: `undef` -##### `r10k_known_hosts` +##### `r10k_known_hosts` Data type: `Optional[Peadm::Known_hosts]` -Default value: ``undef`` +Default value: `undef` -##### `deploy_environment` +##### `deploy_environment` Data type: `Optional[String]` -Default value: ``undef`` +Default value: `undef` -##### `license_key_file` +##### `license_key_file` Data type: `Optional[String]` -Default value: ``undef`` +Default value: `undef` -##### `license_key_content` +##### `license_key_content` Data type: `Optional[String]` -Default value: ``undef`` +Default value: `undef` -##### `download_mode` +##### `download_mode` Data type: `Enum['direct', 'bolthost']` @@ -1923,15 +1918,15 @@ Data type: `Enum['direct', 'bolthost']` Default value: `'bolthost'` -##### `permit_unsafe_versions` +##### `permit_unsafe_versions` Data type: `Boolean` -Default value: ``false`` +Default value: `false` -##### `token_lifetime` +##### `token_lifetime` Data type: `String` @@ -1939,7 +1934,7 @@ Data type: `String` Default value: `'1y'` -### `peadm::modify_certificate` +### `peadm::modify_certificate` Certificates can be modified by adding extensions, removing extensions, or setting DNS alternative names. @@ -1948,26 +1943,26 @@ setting DNS alternative names. The following parameters are available in the `peadm::modify_certificate` plan: -* [`targets`](#targets) -* [`primary_host`](#primary_host) -* [`add_extensions`](#add_extensions) -* [`remove_extensions`](#remove_extensions) -* [`dns_alt_names`](#dns_alt_names) -* [`force_regenerate`](#force_regenerate) +* [`targets`](#-peadm--modify_certificate--targets) +* [`primary_host`](#-peadm--modify_certificate--primary_host) +* [`add_extensions`](#-peadm--modify_certificate--add_extensions) +* [`remove_extensions`](#-peadm--modify_certificate--remove_extensions) +* [`dns_alt_names`](#-peadm--modify_certificate--dns_alt_names) +* [`force_regenerate`](#-peadm--modify_certificate--force_regenerate) -##### `targets` +##### `targets` Data type: `TargetSpec` -##### `primary_host` +##### `primary_host` Data type: `Peadm::SingleTargetSpec` -##### `add_extensions` +##### `add_extensions` Data type: `Hash` @@ -1975,7 +1970,7 @@ Data type: `Hash` Default value: `{}` -##### `remove_extensions` +##### `remove_extensions` Data type: `Array` @@ -1983,23 +1978,23 @@ Data type: `Array` Default value: `[]` -##### `dns_alt_names` +##### `dns_alt_names` Data type: `Optional[Array]` -Default value: ``undef`` +Default value: `undef` -##### `force_regenerate` +##### `force_regenerate` Data type: `Boolean` -Default value: ``false`` +Default value: `false` -### `peadm::restore_ca` +### `peadm::restore_ca` The peadm::restore_ca class. @@ -2007,23 +2002,23 @@ The peadm::restore_ca class. The following parameters are available in the `peadm::restore_ca` plan: -* [`target`](#target) -* [`file_path`](#file_path) -* [`recovery_directory`](#recovery_directory) +* [`target`](#-peadm--restore_ca--target) +* [`file_path`](#-peadm--restore_ca--file_path) +* [`recovery_directory`](#-peadm--restore_ca--recovery_directory) -##### `target` +##### `target` Data type: `Peadm::SingleTargetSpec` -##### `file_path` +##### `file_path` Data type: `String` -##### `recovery_directory` +##### `recovery_directory` Data type: `Optional[String]` @@ -2031,7 +2026,7 @@ Data type: `Optional[String]` Default value: `'/tmp/peadm_recovery'` -### `peadm::status` +### `peadm::status` Return status information from one or more PE clusters in a table format @@ -2047,19 +2042,19 @@ peadm::status($targets, 'table', true, true) The following parameters are available in the `peadm::status` plan: -* [`targets`](#targets) -* [`format`](#format) -* [`summarize`](#summarize) -* [`verbose`](#verbose) -* [`colors`](#colors) +* [`targets`](#-peadm--status--targets) +* [`format`](#-peadm--status--format) +* [`summarize`](#-peadm--status--summarize) +* [`verbose`](#-peadm--status--verbose) +* [`colors`](#-peadm--status--colors) -##### `targets` +##### `targets` Data type: `TargetSpec` These are a list of the primary puppetservers from one or multiple puppet stacks -##### `format` +##### `format` Data type: `Enum[json,table]` @@ -2067,31 +2062,31 @@ The output format to dump to stdout (json or table) Default value: `'table'` -##### `summarize` +##### `summarize` Data type: `Boolean` Controls the type of json output to render, defaults to true -Default value: ``true`` +Default value: `true` -##### `verbose` +##### `verbose` Data type: `Boolean` Toggles the output to show all the operationally services, can be loads more data -Default value: ``false`` +Default value: `false` -##### `colors` +##### `colors` Data type: `Boolean` Toggles the usage of colors, you may want to disable if the format is json -Default value: `$format` +Default value: `$format ? { 'json' => false, default => true` -### `peadm::upgrade` +### `peadm::upgrade` Upgrade a PEAdm-managed cluster @@ -2099,35 +2094,35 @@ Upgrade a PEAdm-managed cluster The following parameters are available in the `peadm::upgrade` plan: -* [`compiler_pool_address`](#compiler_pool_address) -* [`internal_compiler_a_pool_address`](#internal_compiler_a_pool_address) -* [`internal_compiler_b_pool_address`](#internal_compiler_b_pool_address) -* [`pe_installer_source`](#pe_installer_source) -* [`final_agent_state`](#final_agent_state) -* [`r10k_known_hosts`](#r10k_known_hosts) -* [`stagingdir`](#stagingdir) -* [`uploaddir`](#uploaddir) -* [`primary_host`](#primary_host) -* [`replica_host`](#replica_host) -* [`compiler_hosts`](#compiler_hosts) -* [`primary_postgresql_host`](#primary_postgresql_host) -* [`replica_postgresql_host`](#replica_postgresql_host) -* [`version`](#version) -* [`token_file`](#token_file) -* [`download_mode`](#download_mode) -* [`permit_unsafe_versions`](#permit_unsafe_versions) -* [`begin_at_step`](#begin_at_step) - -##### `compiler_pool_address` +* [`compiler_pool_address`](#-peadm--upgrade--compiler_pool_address) +* [`internal_compiler_a_pool_address`](#-peadm--upgrade--internal_compiler_a_pool_address) +* [`internal_compiler_b_pool_address`](#-peadm--upgrade--internal_compiler_b_pool_address) +* [`pe_installer_source`](#-peadm--upgrade--pe_installer_source) +* [`final_agent_state`](#-peadm--upgrade--final_agent_state) +* [`r10k_known_hosts`](#-peadm--upgrade--r10k_known_hosts) +* [`stagingdir`](#-peadm--upgrade--stagingdir) +* [`uploaddir`](#-peadm--upgrade--uploaddir) +* [`primary_host`](#-peadm--upgrade--primary_host) +* [`replica_host`](#-peadm--upgrade--replica_host) +* [`compiler_hosts`](#-peadm--upgrade--compiler_hosts) +* [`primary_postgresql_host`](#-peadm--upgrade--primary_postgresql_host) +* [`replica_postgresql_host`](#-peadm--upgrade--replica_postgresql_host) +* [`version`](#-peadm--upgrade--version) +* [`token_file`](#-peadm--upgrade--token_file) +* [`download_mode`](#-peadm--upgrade--download_mode) +* [`permit_unsafe_versions`](#-peadm--upgrade--permit_unsafe_versions) +* [`begin_at_step`](#-peadm--upgrade--begin_at_step) + +##### `compiler_pool_address` Data type: `Optional[String]` The service address used by agents to connect to compilers, or the Puppet service. Typically this is a load balancer. -Default value: ``undef`` +Default value: `undef` -##### `internal_compiler_a_pool_address` +##### `internal_compiler_a_pool_address` Data type: `Optional[String]` @@ -2135,9 +2130,9 @@ A load balancer address directing traffic to any of the "A" pool compilers. This is used for DR configuration in large and extra large architectures. -Default value: ``undef`` +Default value: `undef` -##### `internal_compiler_b_pool_address` +##### `internal_compiler_b_pool_address` Data type: `Optional[String]` @@ -2145,9 +2140,9 @@ A load balancer address directing traffic to any of the "B" pool compilers. This is used for DR configuration in large and extra large architectures. -Default value: ``undef`` +Default value: `undef` -##### `pe_installer_source` +##### `pe_installer_source` Data type: `Optional[String]` @@ -2156,9 +2151,9 @@ specified, PEAdm will attempt to download PE installation media from its standard public source. When specified, PEAdm will download directly from the URL given. -Default value: ``undef`` +Default value: `undef` -##### `final_agent_state` +##### `final_agent_state` Data type: `Enum['running', 'stopped']` @@ -2167,7 +2162,7 @@ after PE is upgraded successfully. Default value: `'running'` -##### `r10k_known_hosts` +##### `r10k_known_hosts` Data type: `Optional[Peadm::Known_hosts]` @@ -2176,9 +2171,9 @@ r10k_remote host when using ssh. you must provide \$r10k_known_hosts information in the form of an array of hashes with 'name', 'type' and 'key' information for hostname, key-type and public key. -Default value: ``undef`` +Default value: `undef` -##### `stagingdir` +##### `stagingdir` Data type: `String` @@ -2187,7 +2182,7 @@ download_mode is 'bolthost' (default) Default value: `'/tmp'` -##### `uploaddir` +##### `uploaddir` Data type: `String` @@ -2196,61 +2191,61 @@ for offline usage. Default value: `'/tmp'` -##### `primary_host` +##### `primary_host` Data type: `Peadm::SingleTargetSpec` -##### `replica_host` +##### `replica_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `compiler_hosts` +##### `compiler_hosts` Data type: `Optional[TargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `primary_postgresql_host` +##### `primary_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `replica_postgresql_host` +##### `replica_postgresql_host` Data type: `Optional[Peadm::SingleTargetSpec]` -Default value: ``undef`` +Default value: `undef` -##### `version` +##### `version` Data type: `Optional[Peadm::Pe_version]` -Default value: ``undef`` +Default value: `undef` -##### `token_file` +##### `token_file` Data type: `Optional[String]` -Default value: ``undef`` +Default value: `undef` -##### `download_mode` +##### `download_mode` Data type: `Enum[direct,bolthost]` @@ -2258,25 +2253,29 @@ Data type: `Enum[direct,bolthost]` Default value: `'bolthost'` -##### `permit_unsafe_versions` +##### `permit_unsafe_versions` Data type: `Boolean` -Default value: ``false`` +Default value: `false` -##### `begin_at_step` +##### `begin_at_step` -Data type: `Optional[Enum[ +Data type: + +```puppet +Optional[Enum[ 'upgrade-primary', 'upgrade-node-groups', 'upgrade-primary-compilers', 'upgrade-replica', 'upgrade-replica-compilers', - 'finalize']]` + 'finalize']] +``` -Default value: ``undef`` +Default value: `undef` From b368a7ab105dc6f8ea4ba0df404ca7bbb1ed0a9f Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 24 Nov 2023 13:29:56 +0100 Subject: [PATCH 027/110] Change params to support recovery use case --- functions/recovery_opts_default.pp | 12 +++++++----- types/recovery_opts.pp | 8 +++++--- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/functions/recovery_opts_default.pp b/functions/recovery_opts_default.pp index ef5fdfae..5e4b2f4d 100644 --- a/functions/recovery_opts_default.pp +++ b/functions/recovery_opts_default.pp @@ -1,10 +1,12 @@ function peadm::recovery_opts_default () { { - 'orchestrator' => true, + 'activity' => false, + 'ca' => true, + 'classifier' => false, + 'code' => true, + 'config' => true, + 'orchestrator' => false, 'puppetdb' => true, - 'rbac' => true, - 'activity' => true, - 'ca' => false, - 'classifier' => true, + 'rbac' => false, } } diff --git a/types/recovery_opts.pp b/types/recovery_opts.pp index 03b318db..eb392752 100644 --- a/types/recovery_opts.pp +++ b/types/recovery_opts.pp @@ -1,8 +1,10 @@ type Peadm::Recovery_opts = Struct[{ - 'orchestrator' => Optional[Boolean], - 'puppetdb' => Optional[Boolean], - 'rbac' => Optional[Boolean], 'activity' => Optional[Boolean], 'ca' => Optional[Boolean], 'classifier' => Optional[Boolean], + 'code' => Optional[Boolean], + 'config' => Optional[Boolean], + 'orchestrator' => Optional[Boolean], + 'puppetdb' => Optional[Boolean], + 'rbac' => Optional[Boolean], }] From c0d73d83f75b23d2217fbc673fcd5d740708f2bd Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 24 Nov 2023 13:30:52 +0100 Subject: [PATCH 028/110] restore config from backup if primary degrated --- plans/backup.pp | 36 ++++++++++++++++++++++++++++++++++-- plans/restore.pp | 47 ++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 76 insertions(+), 7 deletions(-) diff --git a/plans/backup.pp b/plans/backup.pp index b653e99f..5d167f0a 100644 --- a/plans/backup.pp +++ b/plans/backup.pp @@ -49,7 +49,23 @@ ensure => 'directory', owner => 'root', group => 'root', - mode => '0700', + mode => '0711', + } + + # create a backup subdir for peadm configration + file { "${backup_directory}/peadm": + ensure => 'directory', + owner => 'root', + group => 'root', + mode => '0711', + } + + # backup the cluster config + file { "${backup_directory}/peadm/peadm_config.json": + content => stdlib::to_json_pretty($cluster), + owner => 'root', + group => 'root', + mode => '0600', } # Create a subdir for each backup type selected @@ -58,7 +74,7 @@ ensure => 'directory', owner => 'root', group => 'root', - mode => '0700', + mode => '0711', } } } @@ -78,6 +94,22 @@ | CMD } + if getvar('recovery_opts.code') { + out::message('# Backing up code') + # run_command("chown pe-postgres ${shellquote($backup_directory)}/code", $primary_target) + run_command(@("CMD"), $primary_target) + /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/code --scope=code + | CMD + } + + if getvar('recovery_opts.config') { + out::message('# Backing up config') + run_command("chown pe-postgres ${shellquote($backup_directory)}/config", $primary_target) + run_command(@("CMD"), $primary_target) + /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/config --scope=config + | CMD + } + # Check if /etc/puppetlabs/console-services/conf.d/secrets/keys.json exists and if so back it up if getvar('recovery_opts.rbac') { out::message('# Backing up ldap secret key if it exists') diff --git a/plans/restore.pp b/plans/restore.pp index 6aa74c81..d6b32f6d 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -12,11 +12,27 @@ # Path to the recovery tarball Pattern[/.*\.tar\.gz$/] $input_file, + + # Do we want to merge existing puppetdb content to the restored content? + Boolean $merge_puppetdb = false, ) { peadm::assert_supported_bolt_version() $recovery_opts = (peadm::recovery_opts_default() + $restore) - $cluster = run_task('peadm::get_peadm_config', $targets).first.value + $recovery_directory = "${dirname($input_file)}/${basename($input_file, '.tar.gz')}" + + # try to load the cluster configuration by running peadm::get_peadm_config, but allow for errors to happen + $_cluster = run_task('peadm::get_peadm_config', $targets, { '_catch_errors' => true }).first.value + if $_cluster == undef { + # failed to get cluster config, load from backup + out::message('Failed to get cluster configuration, loading from backup...') + $result = download_file("${recovery_directory}/peadm/peadm_config.json", 'peadm_config.json', $targets).first.value + $cluster = loadjson(getvar('result.path')) + out::message('Cluster configuration loadad from backup') + } else { + $cluster = $_cluster + } + $arch = peadm::assert_supported_architecture( getvar('cluster.params.primary_host'), getvar('cluster.params.replica_host'), @@ -49,7 +65,6 @@ $compiler_targets, ]) - $recovery_directory = "${dirname($input_file)}/${basename($input_file, '.tar.gz')}" # lint:ignore:strict_indent run_command(@("CMD"/L), $primary_target) umask 0077 \ @@ -96,11 +111,33 @@ | CMD } + if getvar('recovery_opts.code') { + out::message('# Restoring code') + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/bin/puppet-backup restore \ + --scope=code \ + --tempdir=${shellquote($recovery_directory)} \ + --force \ + ${shellquote($recovery_directory)}/code/pe_backup-*tgz + | CMD + } + + if getvar('recovery_opts.config') { + out::message('# Restoring config') + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/bin/puppet-backup restore \ + --scope=config \ + --tempdir=${shellquote($recovery_directory)} \ + --force \ + ${shellquote($recovery_directory)}/config/pe_backup-*tgz + | CMD + } + # Use PuppetDB's /pdb/admin/v1/archive API to SAVE data currently in PuppetDB. # Otherwise we'll completely lose it if/when we restore. # TODO: consider adding a heuristic to skip when innappropriate due to size # or other factors. - if getvar('recovery_opts.puppetdb') { + if getvar('recovery_opts.puppetdb') and $merge_puppetdb { out::message('# Exporting puppetdb') run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/bin/puppet-db export \ @@ -121,7 +158,7 @@ run_command(@("CMD"/L), $primary_target) test -f ${shellquote($recovery_directory)}/rbac/keys.json \ && cp -rp ${shellquote($recovery_directory)}/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ \ - || echo secret ldap key doesnt exist + || echo secret ldap key doesn't exist | CMD # lint:ignore:140chars # IF restoring orchestrator restore the secrets to /etc/puppetlabs/orchestration-services/conf.d/secrets/ @@ -220,7 +257,7 @@ # into the restored database. # TODO: consider adding a heuristic to skip when innappropriate due to size # or other factors. - if getvar('recovery_opts.puppetdb') { + if getvar('recovery_opts.puppetdb') and $merge_puppetdb { run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/bin/puppet-db import \ --cert=$(/opt/puppetlabs/bin/puppet config print hostcert) \ From 3f19b4ce176f4342b928c0e09c7bfc8b716bb564 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 24 Nov 2023 14:32:50 +0100 Subject: [PATCH 029/110] fix quoting error --- plans/backup.pp | 2 ++ plans/restore.pp | 7 +++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/plans/backup.pp b/plans/backup.pp index 5d167f0a..f7d33f81 100644 --- a/plans/backup.pp +++ b/plans/backup.pp @@ -3,6 +3,8 @@ # # This plan can backup data as outlined at insert doc # +# TODO +# - make sure backup warns and stops if run on a non-peadm-compatible cluster plan peadm::backup ( # This plan should be run on the primary server Peadm::SingleTargetSpec $targets, diff --git a/plans/restore.pp b/plans/restore.pp index d6b32f6d..a7fa8ca0 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -3,6 +3,9 @@ # # This plan can restore data to puppet infrastructure for DR and rebuilds # +# TODO +# - make sure restore tries to leave the system in a running state if possible +# - merge the restore of certs, config and code scope to one puppet-backup restore command plan peadm::restore ( # This plan should be run on the primary server Peadm::SingleTargetSpec $targets, @@ -28,7 +31,7 @@ out::message('Failed to get cluster configuration, loading from backup...') $result = download_file("${recovery_directory}/peadm/peadm_config.json", 'peadm_config.json', $targets).first.value $cluster = loadjson(getvar('result.path')) - out::message('Cluster configuration loadad from backup') + out::message('Cluster configuration loaded from backup') } else { $cluster = $_cluster } @@ -158,7 +161,7 @@ run_command(@("CMD"/L), $primary_target) test -f ${shellquote($recovery_directory)}/rbac/keys.json \ && cp -rp ${shellquote($recovery_directory)}/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ \ - || echo secret ldap key doesn't exist + || echo secret ldap key doesn\'t exist | CMD # lint:ignore:140chars # IF restoring orchestrator restore the secrets to /etc/puppetlabs/orchestration-services/conf.d/secrets/ From 208f809cfe9f1ee1c9ea680c219ade9162083e58 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 24 Nov 2023 15:40:25 +0100 Subject: [PATCH 030/110] check if backup or restore run on a non-peadm cluster --- plans/backup.pp | 6 ++++-- plans/restore.pp | 5 +++++ tasks/get_peadm_config.rb | 8 +++++++- 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/plans/backup.pp b/plans/backup.pp index f7d33f81..cc91a2fd 100644 --- a/plans/backup.pp +++ b/plans/backup.pp @@ -3,8 +3,6 @@ # # This plan can backup data as outlined at insert doc # -# TODO -# - make sure backup warns and stops if run on a non-peadm-compatible cluster plan peadm::backup ( # This plan should be run on the primary server Peadm::SingleTargetSpec $targets, @@ -19,6 +17,10 @@ $recovery_opts = (peadm::recovery_opts_default() + $backup) $cluster = run_task('peadm::get_peadm_config', $targets).first.value + $error = getvar('cluster.error') + if $error { + fail_plan($error) + } $arch = peadm::assert_supported_architecture( getvar('cluster.params.primary_host'), getvar('cluster.params.replica_host'), diff --git a/plans/restore.pp b/plans/restore.pp index a7fa8ca0..850aa356 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -36,6 +36,11 @@ $cluster = $_cluster } + $error = getvar('cluster.error') + if $error { + fail_plan($error) + } + $arch = peadm::assert_supported_architecture( getvar('cluster.params.primary_host'), getvar('cluster.params.replica_host'), diff --git a/tasks/get_peadm_config.rb b/tasks/get_peadm_config.rb index 5fc3dc56..9bf0993f 100755 --- a/tasks/get_peadm_config.rb +++ b/tasks/get_peadm_config.rb @@ -11,7 +11,13 @@ class GetPEAdmConfig def initialize(params); end def execute! - puts config.to_json + # if there is no 'PE HA Replica' node group, it's not a peadm-configured cluster. + replica_group = groups.data.find { |obj| obj['name'] == 'PE HA Replica' } + if replica_group + puts config.to_json + else + puts ({ 'error' => 'This is not a peadm-compatible cluster. Use peadm::convert first.' }).to_json + end end def config From 88ab524550877b571ff9d61e22e1a4e3f10db86a Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 28 Nov 2023 15:34:56 +0100 Subject: [PATCH 031/110] Add backup_type and restore_type params backup_type and restore_type are "master switches" that select backup and restore options for both scenarios. Fix the bug where the restore broke when classifier service was down. --- functions/migration_opts_default.pp | 12 ++++++++++++ functions/recovery_opts_all.pp | 12 ++++++++++++ plans/backup.pp | 10 +++++++++- plans/restore.pp | 15 ++++++++++++--- 4 files changed, 45 insertions(+), 4 deletions(-) create mode 100644 functions/migration_opts_default.pp create mode 100644 functions/recovery_opts_all.pp diff --git a/functions/migration_opts_default.pp b/functions/migration_opts_default.pp new file mode 100644 index 00000000..99b57e55 --- /dev/null +++ b/functions/migration_opts_default.pp @@ -0,0 +1,12 @@ +function peadm::migration_opts_default () { + { + 'activity' => true, + 'ca' => true, + 'classifier' => true, + 'code' => false, + 'config' => false, + 'orchestrator' => true, + 'puppetdb' => true, + 'rbac' => true, + } +} diff --git a/functions/recovery_opts_all.pp b/functions/recovery_opts_all.pp new file mode 100644 index 00000000..e738ab5e --- /dev/null +++ b/functions/recovery_opts_all.pp @@ -0,0 +1,12 @@ +function peadm::recovery_opts_all () { + { + 'activity' => true, + 'ca' => true, + 'classifier' => true, + 'code' => true, + 'config' => true, + 'orchestrator' => true, + 'puppetdb' => true, + 'rbac' => true, + } +} diff --git a/plans/backup.pp b/plans/backup.pp index cc91a2fd..6660176f 100644 --- a/plans/backup.pp +++ b/plans/backup.pp @@ -7,6 +7,9 @@ # This plan should be run on the primary server Peadm::SingleTargetSpec $targets, + # backup type determines the backup options + Enum['recovery', 'migration', 'custom'] $backup_type = 'recovery', + # Which data to backup Peadm::Recovery_opts $backup = {}, @@ -15,7 +18,6 @@ ) { peadm::assert_supported_bolt_version() - $recovery_opts = (peadm::recovery_opts_default() + $backup) $cluster = run_task('peadm::get_peadm_config', $targets).first.value $error = getvar('cluster.error') if $error { @@ -29,6 +31,12 @@ getvar('cluster.params.compiler_hosts'), ) + $recovery_opts = $backup_type? { + 'recovery' => peadm::recovery_opts_default(), + 'migration' => peadm::migration_opts_default(), + 'custom' => peadm::migration_opts_all() + $backup, + } + $timestamp = Timestamp.new().strftime('%Y-%m-%dT%H%M%SZ') $backup_directory = "${output_directory}/pe-backup-${timestamp}" diff --git a/plans/restore.pp b/plans/restore.pp index 850aa356..de198920 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -5,11 +5,14 @@ # # TODO # - make sure restore tries to leave the system in a running state if possible -# - merge the restore of certs, config and code scope to one puppet-backup restore command +# - potentially merge the restore of certs, config and code scope to one puppet-backup restore command plan peadm::restore ( # This plan should be run on the primary server Peadm::SingleTargetSpec $targets, + # restore type determines the restore options + Enum['recovery', 'migration', 'custom'] $restore_type = 'recovery', + # Which data to restore Peadm::Recovery_opts $restore = {}, @@ -21,12 +24,12 @@ ) { peadm::assert_supported_bolt_version() - $recovery_opts = (peadm::recovery_opts_default() + $restore) $recovery_directory = "${dirname($input_file)}/${basename($input_file, '.tar.gz')}" # try to load the cluster configuration by running peadm::get_peadm_config, but allow for errors to happen $_cluster = run_task('peadm::get_peadm_config', $targets, { '_catch_errors' => true }).first.value - if $_cluster == undef { + out::message("cluster: ${_cluster}") + if $_cluster == undef or getvar('_cluster.params') == undef { # failed to get cluster config, load from backup out::message('Failed to get cluster configuration, loading from backup...') $result = download_file("${recovery_directory}/peadm/peadm_config.json", 'peadm_config.json', $targets).first.value @@ -49,6 +52,12 @@ getvar('cluster.params.compiler_hosts'), ) + $recovery_opts = $restore_type? { + 'recovery' => peadm::recovery_opts_default(), + 'migration' => peadm::migration_opts_default(), + 'custom' => peadm::migration_opts_all() + $restore, + } + $primary_target = peadm::get_targets(getvar('cluster.params.primary_host'), 1) $replica_target = peadm::get_targets(getvar('cluster.params.replica_host'), 1) $compiler_targets = peadm::get_targets(getvar('cluster.params.compiler_hosts')) From a3311038a1471606a6e1d67e232dd0cd682ecff8 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Tue, 28 Nov 2023 15:51:13 +0000 Subject: [PATCH 032/110] Adding test for backup restore Only supporting recovery scenario --- .github/workflows/test-backup-restore.yaml | 132 +++++------------- .../peadm_spec/plans/test_backup.pp | 17 ++- .../peadm_spec/plans/test_restore.pp | 27 ++++ 3 files changed, 72 insertions(+), 104 deletions(-) create mode 100644 spec/acceptance/peadm_spec/plans/test_restore.pp diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 206844f3..ba0c8c42 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -15,7 +15,7 @@ on: version: description: "PE version to install" required: true - default: "2021.7.4" + default: "2023.5.0" ssh-debugging: description: "Boolean; whether or not to pause for ssh debugging" required: true @@ -81,12 +81,6 @@ jobs: sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true echo ::endgroup:: - # - name: Save inventory file A to an artifact - # uses: actions/upload-artifact@v3 - # with: - # name: inventory_A - # path: spec/fixtures/litmus_inventory.yaml - - name: "Install PE on test cluster" timeout-minutes: 120 run: | @@ -96,20 +90,7 @@ jobs: architecture=${{ inputs.architecture }} \ version=${{ inputs.version }} - - name: "Start SSH session" - if: github.event.inputs.ssh-debugging == 'true' - uses: luchihoratiu/debug-via-ssh@main - with: - NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} - SSH_PASS: ${{ secrets.SSH_PASS }} - - # - name: Download artifacts - # # if: always() - # uses: actions/download-artifact@v3 - # with: - # path: spec/fixtures/ - - - name: perform PE backup of cluster A + - name: Perform PE backup of cluster timeout-minutes: 10 continue-on-error: true run: | @@ -125,7 +106,7 @@ jobs: bundle exec bolt plan run peadm_spec::test_backup \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules - echo ::endgroup:: + echo ::endgroup:: - name: "Wait as long as the file ${HOME}/pause file is present" continue-on-error: true @@ -138,46 +119,24 @@ jobs: done echo "${HOME}/pause absent, continuing workflow." - - name: "Tear down cluster A" - if: always() - run: | - if [ -f spec/fixtures/litmus_inventory.yaml ]; then - echo ::group::tear_down - bundle exec rake 'litmus:tear_down' - echo ::endgroup:: - - echo ::group::info:request - cat request.json || true; echo - echo ::endgroup:: - fi - - restore: - name: "Restore: Cluster B: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" - runs-on: ubuntu-20.04 - env: - BOLT_GEM: true - BOLT_DISABLE_ANALYTICS: true - LANG: "en_US.UTF-8" - - steps: - - name: "Checkout Source" - uses: actions/checkout@v2 - - - name: "Activate Ruby 2.7" - uses: ruby/setup-ruby@v1 + - name: Set up yq + uses: frenck/action-setup-yq@v1 with: - ruby-version: "2.7" - bundler-cache: true + version: v4.30.5 - - name: "Print bundle environment" - if: ${{ github.repository_owner == 'puppetlabs' }} + - name: Break PE cluster run: | - echo ::group::info:bundler - bundle env - echo ::endgroup:: + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary" | .uri)' spec/fixtures/litmus_inventory.yaml) + echo "Removing ssl directories" + bundle exec bolt command run "rm -rf /etc/puppetlabs/puppetserver/ca /etc/puppetlabs/puppet/ssl" -t $primary + echo "Removing config file" + bundle exec bolt command run "rm /etc/puppetlabs/enterprise/conf.d/pe.conf" -t $primary - - name: "Provision test cluster" - timeout-minutes: 15 + bundle exec bolt command run "puppet infrastructure status" -t $primary + + - name: Perform PE restore of cluster + timeout-minutes: 10 + continue-on-error: true run: | echo ::group::prepare mkdir -p $HOME/.ssh @@ -187,49 +146,32 @@ jobs: bundle exec rake spec_prep echo ::endgroup:: - echo ::group::provision - bundle exec bolt plan run peadm_spec::provision_test_cluster \ - --modulepath spec/fixtures/modules \ - provider=provision_service \ - image=${{ inputs.image }} \ - architecture=${{ inputs.architecture }} - echo ::endgroup:: - - echo ::group::info:request - cat request.json || true; echo - echo ::endgroup:: - - echo ::group::info:inventory - sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true + echo ::group::restore + TARBALL=$(echo /tmp/pe-backup*gz) + bundle exec bolt plan run peadm_spec::test_restore input_file="$TARBALL" \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules echo ::endgroup:: - # - name: Save inventory file B to an artifact - # uses: actions/upload-artifact@v3 - # with: - # name: inventory_B - # path: spec/fixtures/litmus_inventory.yaml - - - name: "Install PE on test cluster" - timeout-minutes: 120 + - name: Output PE cluster status run: | - bundle exec bolt plan run peadm_spec::install_test_cluster \ - --inventoryfile spec/fixtures/litmus_inventory.yaml \ - --modulepath spec/fixtures/modules \ - architecture=${{ inputs.architecture }} \ - version=${{ inputs.version }} - - - name: Wait for backup to finish - uses: lewagon/wait-on-check-action@v1.3.1 - with: - ref: ${{ github.ref }} - check-name: "Backup: Cluster A: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" - repo-token: ${{ secrets.GITHUB_TOKEN }} - wait-interval: 10 + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary" | .uri)' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt command run "puppet infrastructure status" -t $primary + + - name: "Wait as long as the file ${HOME}/pause2 file is present" + continue-on-error: true + # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} + if: github.event.inputs.ssh-debugging == 'true' + run: | + while [ -f "${HOME}/pause2" ] ; do + echo "${HOME}/pause2 present, sleeping for 60 seconds..." + sleep 10 + done + echo "${HOME}/pause2 absent, continuing workflow." - - name: "Tear down cluster B" + - name: "Tear down cluster" if: always() run: | - cp spec/fixtures/inventory_B/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true if [ -f spec/fixtures/litmus_inventory.yaml ]; then echo ::group::tear_down bundle exec rake 'litmus:tear_down' diff --git a/spec/acceptance/peadm_spec/plans/test_backup.pp b/spec/acceptance/peadm_spec/plans/test_backup.pp index 9c02eeb2..347f9a99 100644 --- a/spec/acceptance/peadm_spec/plans/test_backup.pp +++ b/spec/acceptance/peadm_spec/plans/test_backup.pp @@ -10,15 +10,14 @@ # run infra status on the primary $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] out::message("Running peadm::status on primary host ${primary_host}") - run_plan('peadm::status', $primary_host) + $result = run_plan('peadm::status', $primary_host, { 'format' => 'json' }) - $backup_options = { - 'orchestrator' => true, - 'puppetdb' => true, - 'rbac' => true, - 'activity' => true, - 'ca' => true, - 'classifier' => true, + out::message($result.first.value) + + if $result.first.value['failed'] == undef { # empty array + out::message('Cluster is healthy, continuing') + } else { + fail_plan('Cluster is not healthy, aborting') } - run_plan('peadm::backup', $primary_host, { 'output_directory' => '/tmp', 'backup' => $backup_options }) + run_plan('peadm::backup', $primary_host, { 'output_directory' => '/tmp', 'backup_type' => 'recovery' }) } diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp new file mode 100644 index 00000000..49611b3e --- /dev/null +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -0,0 +1,27 @@ +plan peadm_spec::test_restore( + String[1] $input_file, +) { + $t = get_targets('*') + wait_until_available($t) + + parallelize($t) |$target| { + $fqdn = run_command('hostname -f', $target) + $target.set_var('certname', $fqdn.first['stdout'].chomp) + } + + $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] + + run_plan('peadm::restore', $primary_host, { 'backup_type' => 'recovery', 'input_file' => $input_file }) + + # run infra status on the primary + out::message("Running peadm::status on primary host ${primary_host}") + $result = run_plan('peadm::status', $primary_host, { 'format' => 'json' }) + + out::message($result.first.value) + + if $result.first.value['failed'] == undef { # empty array + out::message('Cluster is healthy, continuing') + } else { + fail_plan('Cluster is not healthy, aborting') + } +} From 182e6f1ed721891605e410dc09e5ad0b6cd19d74 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Tue, 28 Nov 2023 15:51:13 +0000 Subject: [PATCH 033/110] Adding test for backup restore Only supporting recovery scenario --- .github/workflows/test-backup-restore.yaml | 136 ++++++------------ Gemfile | 2 +- .../peadm_spec/plans/test_backup.pp | 17 ++- .../peadm_spec/plans/test_restore.pp | 27 ++++ 4 files changed, 76 insertions(+), 106 deletions(-) create mode 100644 spec/acceptance/peadm_spec/plans/test_restore.pp diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 206844f3..980c2f25 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -15,7 +15,7 @@ on: version: description: "PE version to install" required: true - default: "2021.7.4" + default: "2023.5.0" ssh-debugging: description: "Boolean; whether or not to pause for ssh debugging" required: true @@ -23,7 +23,7 @@ on: jobs: backup: - name: "Backup: Cluster A: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" + name: "Backup cluster: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" runs-on: ubuntu-20.04 env: BOLT_GEM: true @@ -81,12 +81,6 @@ jobs: sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true echo ::endgroup:: - # - name: Save inventory file A to an artifact - # uses: actions/upload-artifact@v3 - # with: - # name: inventory_A - # path: spec/fixtures/litmus_inventory.yaml - - name: "Install PE on test cluster" timeout-minutes: 120 run: | @@ -94,22 +88,11 @@ jobs: --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ architecture=${{ inputs.architecture }} \ - version=${{ inputs.version }} - - - name: "Start SSH session" - if: github.event.inputs.ssh-debugging == 'true' - uses: luchihoratiu/debug-via-ssh@main - with: - NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} - SSH_PASS: ${{ secrets.SSH_PASS }} + version=${{ inputs.version }} \ + --stream - # - name: Download artifacts - # # if: always() - # uses: actions/download-artifact@v3 - # with: - # path: spec/fixtures/ - - name: perform PE backup of cluster A + - name: Perform peadm backup of cluster timeout-minutes: 10 continue-on-error: true run: | @@ -125,7 +108,7 @@ jobs: bundle exec bolt plan run peadm_spec::test_backup \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules - echo ::endgroup:: + echo ::endgroup:: - name: "Wait as long as the file ${HOME}/pause file is present" continue-on-error: true @@ -138,46 +121,24 @@ jobs: done echo "${HOME}/pause absent, continuing workflow." - - name: "Tear down cluster A" - if: always() - run: | - if [ -f spec/fixtures/litmus_inventory.yaml ]; then - echo ::group::tear_down - bundle exec rake 'litmus:tear_down' - echo ::endgroup:: - - echo ::group::info:request - cat request.json || true; echo - echo ::endgroup:: - fi - - restore: - name: "Restore: Cluster B: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" - runs-on: ubuntu-20.04 - env: - BOLT_GEM: true - BOLT_DISABLE_ANALYTICS: true - LANG: "en_US.UTF-8" - - steps: - - name: "Checkout Source" - uses: actions/checkout@v2 - - - name: "Activate Ruby 2.7" - uses: ruby/setup-ruby@v1 + - name: Set up yq + uses: frenck/action-setup-yq@v1 with: - ruby-version: "2.7" - bundler-cache: true + version: v4.30.5 - - name: "Print bundle environment" - if: ${{ github.repository_owner == 'puppetlabs' }} + - name: Break PE cluster run: | - echo ::group::info:bundler - bundle env - echo ::endgroup:: + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary" | .uri)' spec/fixtures/litmus_inventory.yaml) + echo "Removing ssl directories" + bundle exec bolt command run "rm -rf /etc/puppetlabs/puppetserver/ca /etc/puppetlabs/puppet/ssl" -t $primary + echo "Removing config file" + bundle exec bolt command run "rm /etc/puppetlabs/enterprise/conf.d/pe.conf" -t $primary - - name: "Provision test cluster" - timeout-minutes: 15 + bundle exec bolt command run "puppet infrastructure status" -t $primary + + - name: Perform peadm restore of cluster + timeout-minutes: 10 + continue-on-error: true run: | echo ::group::prepare mkdir -p $HOME/.ssh @@ -187,49 +148,32 @@ jobs: bundle exec rake spec_prep echo ::endgroup:: - echo ::group::provision - bundle exec bolt plan run peadm_spec::provision_test_cluster \ - --modulepath spec/fixtures/modules \ - provider=provision_service \ - image=${{ inputs.image }} \ - architecture=${{ inputs.architecture }} - echo ::endgroup:: - - echo ::group::info:request - cat request.json || true; echo - echo ::endgroup:: - - echo ::group::info:inventory - sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true + echo ::group::restore + TARBALL=$(echo /tmp/pe-backup*gz) + bundle exec bolt plan run peadm_spec::test_restore input_file="$TARBALL" \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules echo ::endgroup:: - # - name: Save inventory file B to an artifact - # uses: actions/upload-artifact@v3 - # with: - # name: inventory_B - # path: spec/fixtures/litmus_inventory.yaml - - - name: "Install PE on test cluster" - timeout-minutes: 120 + - name: Output PE cluster status run: | - bundle exec bolt plan run peadm_spec::install_test_cluster \ - --inventoryfile spec/fixtures/litmus_inventory.yaml \ - --modulepath spec/fixtures/modules \ - architecture=${{ inputs.architecture }} \ - version=${{ inputs.version }} - - - name: Wait for backup to finish - uses: lewagon/wait-on-check-action@v1.3.1 - with: - ref: ${{ github.ref }} - check-name: "Backup: Cluster A: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" - repo-token: ${{ secrets.GITHUB_TOKEN }} - wait-interval: 10 + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary" | .uri)' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt command run "puppet infrastructure status" -t $primary + + - name: "Wait as long as the file ${HOME}/pause2 file is present" + continue-on-error: true + # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} + if: github.event.inputs.ssh-debugging == 'true' + run: | + while [ -f "${HOME}/pause2" ] ; do + echo "${HOME}/pause2 present, sleeping for 60 seconds..." + sleep 10 + done + echo "${HOME}/pause2 absent, continuing workflow." - - name: "Tear down cluster B" + - name: "Tear down cluster" if: always() run: | - cp spec/fixtures/inventory_B/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true if [ -f spec/fixtures/litmus_inventory.yaml ]; then echo ::group::tear_down bundle exec rake 'litmus:tear_down' diff --git a/Gemfile b/Gemfile index c6c71af2..9322a589 100644 --- a/Gemfile +++ b/Gemfile @@ -35,7 +35,7 @@ group :development do gem "rubocop-performance", '= 1.16.0', require: false gem "rubocop-rspec", '= 2.19.0', require: false gem "rb-readline", '= 0.5.5', require: false, platforms: [:mswin, :mingw, :x64_mingw] - gem "bolt", '>= 3.10.0', require: false + gem "bolt", '>= 3.27.2', require: false gem "github_changelog_generator", '1.16.4', require: false gem "octokit", '4.21.0', require: false end diff --git a/spec/acceptance/peadm_spec/plans/test_backup.pp b/spec/acceptance/peadm_spec/plans/test_backup.pp index 9c02eeb2..110b21cf 100644 --- a/spec/acceptance/peadm_spec/plans/test_backup.pp +++ b/spec/acceptance/peadm_spec/plans/test_backup.pp @@ -10,15 +10,14 @@ # run infra status on the primary $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] out::message("Running peadm::status on primary host ${primary_host}") - run_plan('peadm::status', $primary_host) + $result = run_plan('peadm::status', $primary_host, { 'format' => 'json' }) - $backup_options = { - 'orchestrator' => true, - 'puppetdb' => true, - 'rbac' => true, - 'activity' => true, - 'ca' => true, - 'classifier' => true, + out::message($result) + + if empty($result['failed']) { + out::message('Cluster is healthy, continuing') + } else { + fail_plan('Cluster is not healthy, aborting') } - run_plan('peadm::backup', $primary_host, { 'output_directory' => '/tmp', 'backup' => $backup_options }) + run_plan('peadm::backup', $primary_host, { 'output_directory' => '/tmp', 'backup_type' => 'recovery' }) } diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp new file mode 100644 index 00000000..49611b3e --- /dev/null +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -0,0 +1,27 @@ +plan peadm_spec::test_restore( + String[1] $input_file, +) { + $t = get_targets('*') + wait_until_available($t) + + parallelize($t) |$target| { + $fqdn = run_command('hostname -f', $target) + $target.set_var('certname', $fqdn.first['stdout'].chomp) + } + + $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] + + run_plan('peadm::restore', $primary_host, { 'backup_type' => 'recovery', 'input_file' => $input_file }) + + # run infra status on the primary + out::message("Running peadm::status on primary host ${primary_host}") + $result = run_plan('peadm::status', $primary_host, { 'format' => 'json' }) + + out::message($result.first.value) + + if $result.first.value['failed'] == undef { # empty array + out::message('Cluster is healthy, continuing') + } else { + fail_plan('Cluster is not healthy, aborting') + } +} From b4f87f92f9cd7a5d0b7d0e9c92b030218d55842b Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 28 Nov 2023 18:25:12 +0100 Subject: [PATCH 034/110] Fix issue in test_restore.pp --- spec/acceptance/peadm_spec/plans/test_restore.pp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index 49611b3e..30ce75fa 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -17,9 +17,9 @@ out::message("Running peadm::status on primary host ${primary_host}") $result = run_plan('peadm::status', $primary_host, { 'format' => 'json' }) - out::message($result.first.value) + out::message($result) - if $result.first.value['failed'] == undef { # empty array + if empty($result['failed']) { out::message('Cluster is healthy, continuing') } else { fail_plan('Cluster is not healthy, aborting') From 713a9ea72b4f3d4ee4d389090d12450765b17091 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 28 Nov 2023 18:29:53 +0100 Subject: [PATCH 035/110] fix merge conflict --- .github/workflows/test-backup-restore.yaml | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 1dc333a6..64585a9a 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -107,7 +107,7 @@ jobs: bundle exec bolt plan run peadm_spec::test_backup \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules - echo ::endgroup:: + echo ::endgroup:: - name: "Wait as long as the file ${HOME}/pause file is present" continue-on-error: true @@ -135,11 +135,7 @@ jobs: bundle exec bolt command run "puppet infrastructure status" -t $primary -<<<<<<< HEAD - - name: Perform PE restore of cluster -======= - name: Perform peadm restore of cluster ->>>>>>> 182e6f1ed721891605e410dc09e5ad0b6cd19d74 timeout-minutes: 10 continue-on-error: true run: | @@ -161,8 +157,8 @@ jobs: - name: Output PE cluster status run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary" | .uri)' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt command run "puppet infrastructure status" -t $primary - + bundle exec bolt command run "puppet infrastructure status" -t $primary + - name: "Wait as long as the file ${HOME}/pause2 file is present" continue-on-error: true # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} From 4ad111d8479390a1896ad59f5124bc0206f03dd3 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Tue, 28 Nov 2023 18:07:31 +0000 Subject: [PATCH 036/110] Changed backup to use ip instead of hostname --- plans/backup.pp | 22 +++++++++---------- .../peadm_spec/plans/test_restore.pp | 4 ++-- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/plans/backup.pp b/plans/backup.pp index 6660176f..e38ce48f 100644 --- a/plans/backup.pp +++ b/plans/backup.pp @@ -56,7 +56,7 @@ } # Create backup folders - apply($primary_target) { + apply($targets) { file { $backup_directory : ensure => 'directory', owner => 'root', @@ -93,7 +93,7 @@ if getvar('recovery_opts.classifier') { out::message('# Backing up classification') - run_task('peadm::backup_classification', $primary_target, + run_task('peadm::backup_classification', $targets, directory => "${backup_directory}/classifier", ) } @@ -101,23 +101,23 @@ if getvar('recovery_opts.ca') { out::message('# Backing up ca and ssl certificates') # lint:ignore:strict_indent - run_command(@("CMD"), $primary_target) + run_command(@("CMD"), $targets) /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/ca --scope=certs | CMD } if getvar('recovery_opts.code') { out::message('# Backing up code') - # run_command("chown pe-postgres ${shellquote($backup_directory)}/code", $primary_target) - run_command(@("CMD"), $primary_target) + # run_command("chown pe-postgres ${shellquote($backup_directory)}/code", $targets) + run_command(@("CMD"), $targets) /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/code --scope=code | CMD } if getvar('recovery_opts.config') { out::message('# Backing up config') - run_command("chown pe-postgres ${shellquote($backup_directory)}/config", $primary_target) - run_command(@("CMD"), $primary_target) + run_command("chown pe-postgres ${shellquote($backup_directory)}/config", $targets) + run_command(@("CMD"), $targets) /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/config --scope=config | CMD } @@ -126,7 +126,7 @@ if getvar('recovery_opts.rbac') { out::message('# Backing up ldap secret key if it exists') # lint:ignore:140chars - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) test -f /etc/puppetlabs/console-services/conf.d/secrets/keys.json \ && cp -rp /etc/puppetlabs/console-services/conf.d/secrets ${shellquote($backup_directory)}/rbac/ \ || echo secret ldap key doesnt exist @@ -137,13 +137,13 @@ # IF backing up orchestrator back up the secrets too /etc/puppetlabs/orchestration-services/conf.d/secrets/ if getvar('recovery_opts.orchestrator') { out::message('# Backing up orchestrator secret keys') - run_command(@("CMD"), $primary_target) + run_command(@("CMD"), $targets) cp -rp /etc/puppetlabs/orchestration-services/conf.d/secrets ${shellquote($backup_directory)}/orchestrator/ | CMD } # lint:endignore $backup_databases.each |$name,$database_target| { - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) /opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 \ -f ${shellquote($backup_directory)}/${shellquote($name)}/pe-${shellquote($name)}.dump.d \ "sslmode=verify-ca \ @@ -156,7 +156,7 @@ | CMD } - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) umask 0077 \ && cd ${shellquote(dirname($backup_directory))} \ && tar -czf ${shellquote($backup_directory)}.tar.gz ${shellquote(basename($backup_directory))} \ diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index 49611b3e..30ce75fa 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -17,9 +17,9 @@ out::message("Running peadm::status on primary host ${primary_host}") $result = run_plan('peadm::status', $primary_host, { 'format' => 'json' }) - out::message($result.first.value) + out::message($result) - if $result.first.value['failed'] == undef { # empty array + if empty($result['failed']) { out::message('Cluster is healthy, continuing') } else { fail_plan('Cluster is not healthy, aborting') From 4879fa8c42947a648a7a0430d925d8cd69dacae8 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 28 Nov 2023 20:49:11 +0100 Subject: [PATCH 037/110] Use $targets for running tasks and commands Change plan to use $targets rather than $primary_target to run tasks and commands. --- plans/restore.pp | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/plans/restore.pp b/plans/restore.pp index de198920..1749d0ba 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -83,7 +83,7 @@ ]) # lint:ignore:strict_indent - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) umask 0077 \ && cd ${shellquote(dirname($recovery_directory))} \ && tar -xzf ${shellquote($input_file)} @@ -102,16 +102,16 @@ if getvar('recovery_opts.classifier') { out::message('# Restoring classification') - run_task('peadm::backup_classification', $primary_target, + run_task('peadm::backup_classification', $targets, directory => $recovery_directory ) - run_task('peadm::transform_classification_groups', $primary_target, + run_task('peadm::transform_classification_groups', $targets, source_directory => "${recovery_directory}/classifier", working_directory => $recovery_directory ) - run_task('peadm::restore_classification', $primary_target, + run_task('peadm::restore_classification', $targets, classification_file => "${recovery_directory}/transformed_classification.json", ) } @@ -119,7 +119,7 @@ if getvar('recovery_opts.ca') { out::message('# Restoring ca and ssl certificates') # lint:ignore:strict_indent - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) /opt/puppetlabs/bin/puppet-backup restore \ --scope=certs \ --tempdir=${shellquote($recovery_directory)} \ @@ -130,7 +130,7 @@ if getvar('recovery_opts.code') { out::message('# Restoring code') - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) /opt/puppetlabs/bin/puppet-backup restore \ --scope=code \ --tempdir=${shellquote($recovery_directory)} \ @@ -141,7 +141,7 @@ if getvar('recovery_opts.config') { out::message('# Restoring config') - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) /opt/puppetlabs/bin/puppet-backup restore \ --scope=config \ --tempdir=${shellquote($recovery_directory)} \ @@ -156,7 +156,7 @@ # or other factors. if getvar('recovery_opts.puppetdb') and $merge_puppetdb { out::message('# Exporting puppetdb') - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) /opt/puppetlabs/bin/puppet-db export \ --cert=$(/opt/puppetlabs/bin/puppet config print hostcert) \ --key=$(/opt/puppetlabs/bin/puppet config print hostprivkey) \ @@ -165,14 +165,14 @@ } ## shutdown services - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) systemctl stop pe-console-services pe-nginx pxp-agent pe-puppetserver \ pe-orchestration-services puppet pe-puppetdb | CMD # Restore secrets/keys.json if it exists out::message('# Restoring ldap secret key if it exists') - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) test -f ${shellquote($recovery_directory)}/rbac/keys.json \ && cp -rp ${shellquote($recovery_directory)}/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ \ || echo secret ldap key doesn\'t exist @@ -181,7 +181,7 @@ # IF restoring orchestrator restore the secrets to /etc/puppetlabs/orchestration-services/conf.d/secrets/ if getvar('recovery_opts.orchestrator') { out::message('# Restoring orchestrator secret keys') - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) cp -rp ${shellquote($recovery_directory)}/orchestrator/secrets/* /etc/puppetlabs/orchestration-services/conf.d/secrets/ | CMD } @@ -219,7 +219,7 @@ # Restore database. If there are multiple database restore targets, perform # the restore(s) in parallel. parallelize($database_targets) |$database_target| { - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) /opt/puppetlabs/server/bin/pg_restore \ -j 4 \ -d "sslmode=verify-ca \ @@ -261,7 +261,7 @@ # Use `puppet infra` to ensure correct file permissions, restart services, # etc. Make sure not to try and get config data from the classifier, which # isn't yet up and running. - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) /opt/puppetlabs/bin/puppet-infrastructure configure --no-recover | CMD @@ -275,7 +275,7 @@ # TODO: consider adding a heuristic to skip when innappropriate due to size # or other factors. if getvar('recovery_opts.puppetdb') and $merge_puppetdb { - run_command(@("CMD"/L), $primary_target) + run_command(@("CMD"/L), $targets) /opt/puppetlabs/bin/puppet-db import \ --cert=$(/opt/puppetlabs/bin/puppet config print hostcert) \ --key=$(/opt/puppetlabs/bin/puppet config print hostprivkey) \ @@ -285,9 +285,9 @@ } # Run Puppet to pick up last remaining config tweaks - run_task('peadm::puppet_runonce', $primary_target) + run_task('peadm::puppet_runonce', $targets) - apply($primary_target) { + apply($targets) { file { $recovery_directory : ensure => 'absent', force => true, From da2d2731cb91ca1779ee79c724723b809383e716 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Wed, 29 Nov 2023 10:36:37 +0000 Subject: [PATCH 038/110] Moving pause below yq --- .github/workflows/test-backup-restore.yaml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 64585a9a..6d142657 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -109,6 +109,11 @@ jobs: --modulepath spec/fixtures/modules echo ::endgroup:: + - name: Set up yq + uses: frenck/action-setup-yq@v1 + with: + version: v4.30.5 + - name: "Wait as long as the file ${HOME}/pause file is present" continue-on-error: true # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} @@ -120,11 +125,6 @@ jobs: done echo "${HOME}/pause absent, continuing workflow." - - name: Set up yq - uses: frenck/action-setup-yq@v1 - with: - version: v4.30.5 - - name: Break PE cluster run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary" | .uri)' spec/fixtures/litmus_inventory.yaml) From fdba96bc4cbd1f0c1c8e1af5acc50b9e198c13cf Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Wed, 29 Nov 2023 11:17:32 +0000 Subject: [PATCH 039/110] Moving .uri outside select --- .github/workflows/test-backup-restore.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 6d142657..eab83de9 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -127,7 +127,7 @@ jobs: - name: Break PE cluster run: | - primary=$(yq '.groups[].targets[] | select(.vars.role == "primary" | .uri)' spec/fixtures/litmus_inventory.yaml) + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) echo "Removing ssl directories" bundle exec bolt command run "rm -rf /etc/puppetlabs/puppetserver/ca /etc/puppetlabs/puppet/ssl" -t $primary echo "Removing config file" From c74410c8bc9943deb9247b1e957daf530a465d38 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Wed, 29 Nov 2023 18:33:33 +0000 Subject: [PATCH 040/110] Added inventory file to commands and moving backup file lookup inside test_restore --- .github/workflows/test-backup-restore.yaml | 16 ++++++++-------- spec/acceptance/peadm_spec/plans/test_restore.pp | 8 ++++---- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index eab83de9..7e31e06f 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -129,11 +129,11 @@ jobs: run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) echo "Removing ssl directories" - bundle exec bolt command run "rm -rf /etc/puppetlabs/puppetserver/ca /etc/puppetlabs/puppet/ssl" -t $primary + bundle exec bolt command run "rm -rf /etc/puppetlabs/puppetserver/ca /etc/puppetlabs/puppet/ssl" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml echo "Removing config file" - bundle exec bolt command run "rm /etc/puppetlabs/enterprise/conf.d/pe.conf" -t $primary + bundle exec bolt command run "rm /etc/puppetlabs/enterprise/conf.d/pe.conf" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - bundle exec bolt command run "puppet infrastructure status" -t $primary + bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - name: Perform peadm restore of cluster timeout-minutes: 10 @@ -148,16 +148,16 @@ jobs: echo ::endgroup:: echo ::group::restore - TARBALL=$(echo /tmp/pe-backup*gz) - bundle exec bolt plan run peadm_spec::test_restore input_file="$TARBALL" \ + bundle exec bolt plan run peadm_spec::test_restore \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ - --modulepath spec/fixtures/modules + --modulepath spec/fixtures/modules \ + --stream echo ::endgroup:: - name: Output PE cluster status run: | - primary=$(yq '.groups[].targets[] | select(.vars.role == "primary" | .uri)' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt command run "puppet infrastructure status" -t $primary + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - name: "Wait as long as the file ${HOME}/pause2 file is present" continue-on-error: true diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index 30ce75fa..165c6d8a 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -1,6 +1,4 @@ -plan peadm_spec::test_restore( - String[1] $input_file, -) { +plan peadm_spec::test_restore() { $t = get_targets('*') wait_until_available($t) @@ -11,7 +9,9 @@ $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] - run_plan('peadm::restore', $primary_host, { 'backup_type' => 'recovery', 'input_file' => $input_file }) + input_file=$(ls /tmp/pe-backup*gz) + + run_plan('peadm::restore', $primary_host, { 'restore_type' => 'recovery', 'input_file' => $input_file }) # run infra status on the primary out::message("Running peadm::status on primary host ${primary_host}") From 7050b83900e7f66aeace80bedb90db1dd56e19c8 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Wed, 29 Nov 2023 19:11:48 +0000 Subject: [PATCH 041/110] Fixing backup file location --- spec/acceptance/peadm_spec/plans/test_restore.pp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index 165c6d8a..0cec9625 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -9,7 +9,7 @@ $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] - input_file=$(ls /tmp/pe-backup*gz) + $input_file = run_command('ls /tmp/pe-backup*gz', $primary_host) run_plan('peadm::restore', $primary_host, { 'restore_type' => 'recovery', 'input_file' => $input_file }) From cedb4c14f9abaed8d9562945b51385b7ffa6e884 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Thu, 30 Nov 2023 00:06:10 +0100 Subject: [PATCH 042/110] Fix ls command result --- spec/acceptance/peadm_spec/plans/test_restore.pp | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index 0cec9625..7832837e 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -9,7 +9,9 @@ $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] - $input_file = run_command('ls /tmp/pe-backup*gz', $primary_host) + # get the latest backup file, if more than one exists + $result = run_command('ls -t /tmp/pe-backup*gz | head -1', $primary_host).first.value + $input_file = getvar('result.stdout') run_plan('peadm::restore', $primary_host, { 'restore_type' => 'recovery', 'input_file' => $input_file }) From 7fee7b312df8657b11503d4c361c287cbbb78743 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Thu, 30 Nov 2023 01:14:16 +0100 Subject: [PATCH 043/110] strip leading spaces from tar filename --- spec/acceptance/peadm_spec/plans/test_restore.pp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index 7832837e..b48793c2 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -11,7 +11,7 @@ # get the latest backup file, if more than one exists $result = run_command('ls -t /tmp/pe-backup*gz | head -1', $primary_host).first.value - $input_file = getvar('result.stdout') + $input_file = strip(getvar('result.stdout')) run_plan('peadm::restore', $primary_host, { 'restore_type' => 'recovery', 'input_file' => $input_file }) From 4e7163f3ba2b76648ef97621635f63da62b63ade Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Thu, 30 Nov 2023 01:14:52 +0100 Subject: [PATCH 044/110] untar before using peadm config file --- plans/restore.pp | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/plans/restore.pp b/plans/restore.pp index 1749d0ba..10bfe4de 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -25,6 +25,13 @@ peadm::assert_supported_bolt_version() $recovery_directory = "${dirname($input_file)}/${basename($input_file, '.tar.gz')}" +# lint:ignore:strict_indent + run_command(@("CMD"/L), $targets) + umask 0077 \ + && cd ${shellquote(dirname($recovery_directory))} \ + && tar -xzf ${shellquote($input_file)} + | CMD +# lint:endignore # try to load the cluster configuration by running peadm::get_peadm_config, but allow for errors to happen $_cluster = run_task('peadm::get_peadm_config', $targets, { '_catch_errors' => true }).first.value @@ -82,13 +89,6 @@ $compiler_targets, ]) -# lint:ignore:strict_indent - run_command(@("CMD"/L), $targets) - umask 0077 \ - && cd ${shellquote(dirname($recovery_directory))} \ - && tar -xzf ${shellquote($input_file)} - | CMD -# lint:endignore # Map of recovery option name to array of database hosts to restore the # relevant .dump content to. $restore_databases = { From dff1c239dbdae8d56bf9d662770bae0a76780822 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Thu, 30 Nov 2023 15:22:42 +0100 Subject: [PATCH 045/110] Adding host entries to /etc/hosts --- .github/workflows/test-backup-restore.yaml | 23 +++++-------------- .../peadm_spec/plans/test_restore.pp | 5 +++- 2 files changed, 10 insertions(+), 18 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 7e31e06f..914205b3 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -113,17 +113,6 @@ jobs: uses: frenck/action-setup-yq@v1 with: version: v4.30.5 - - - name: "Wait as long as the file ${HOME}/pause file is present" - continue-on-error: true - # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} - if: github.event.inputs.ssh-debugging == 'true' - run: | - while [ -f "${HOME}/pause" ] ; do - echo "${HOME}/pause present, sleeping for 60 seconds..." - sleep 10 - done - echo "${HOME}/pause absent, continuing workflow." - name: Break PE cluster run: | @@ -159,16 +148,16 @@ jobs: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - - name: "Wait as long as the file ${HOME}/pause2 file is present" + - name: "Wait as long as the file ${HOME}/pause file is present" continue-on-error: true - # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} - if: github.event.inputs.ssh-debugging == 'true' + if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} + # if: github.event.inputs.ssh-debugging == 'true' run: | - while [ -f "${HOME}/pause2" ] ; do - echo "${HOME}/pause2 present, sleeping for 60 seconds..." + while [ -f "${HOME}/pause" ] ; do + echo "${HOME}/pause present, sleeping for 60 seconds..." sleep 10 done - echo "${HOME}/pause2 absent, continuing workflow." + echo "${HOME}/pause absent, continuing workflow." - name: "Tear down cluster" if: always() diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index b48793c2..4fbaab71 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -5,13 +5,16 @@ parallelize($t) |$target| { $fqdn = run_command('hostname -f', $target) $target.set_var('certname', $fqdn.first['stdout'].chomp) + + $command = "echo '${target.vars['certname']} ${target.uri}' | sudo tee -a /etc/hosts" + run_command($command, 'localhost') } $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] # get the latest backup file, if more than one exists $result = run_command('ls -t /tmp/pe-backup*gz | head -1', $primary_host).first.value - $input_file = strip(getvar('result.stdout')) + $input_file = getvar('result.stdout') run_plan('peadm::restore', $primary_host, { 'restore_type' => 'recovery', 'input_file' => $input_file }) From 32513a4fea36f90f1e786c0dc0a43f4f0d09cc41 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Thu, 30 Nov 2023 16:50:26 +0100 Subject: [PATCH 046/110] fix hosts file format revert change of $primary_target to $targets --- plans/restore.pp | 30 +++++++++---------- .../peadm_spec/plans/test_restore.pp | 4 +-- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/plans/restore.pp b/plans/restore.pp index 10bfe4de..9dc1a3ed 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -102,16 +102,16 @@ if getvar('recovery_opts.classifier') { out::message('# Restoring classification') - run_task('peadm::backup_classification', $targets, + run_task('peadm::backup_classification', $primary_target, directory => $recovery_directory ) - run_task('peadm::transform_classification_groups', $targets, + run_task('peadm::transform_classification_groups', $primary_target, source_directory => "${recovery_directory}/classifier", working_directory => $recovery_directory ) - run_task('peadm::restore_classification', $targets, + run_task('peadm::restore_classification', $primary_target, classification_file => "${recovery_directory}/transformed_classification.json", ) } @@ -119,7 +119,7 @@ if getvar('recovery_opts.ca') { out::message('# Restoring ca and ssl certificates') # lint:ignore:strict_indent - run_command(@("CMD"/L), $targets) + run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/bin/puppet-backup restore \ --scope=certs \ --tempdir=${shellquote($recovery_directory)} \ @@ -130,7 +130,7 @@ if getvar('recovery_opts.code') { out::message('# Restoring code') - run_command(@("CMD"/L), $targets) + run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/bin/puppet-backup restore \ --scope=code \ --tempdir=${shellquote($recovery_directory)} \ @@ -141,7 +141,7 @@ if getvar('recovery_opts.config') { out::message('# Restoring config') - run_command(@("CMD"/L), $targets) + run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/bin/puppet-backup restore \ --scope=config \ --tempdir=${shellquote($recovery_directory)} \ @@ -156,7 +156,7 @@ # or other factors. if getvar('recovery_opts.puppetdb') and $merge_puppetdb { out::message('# Exporting puppetdb') - run_command(@("CMD"/L), $targets) + run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/bin/puppet-db export \ --cert=$(/opt/puppetlabs/bin/puppet config print hostcert) \ --key=$(/opt/puppetlabs/bin/puppet config print hostprivkey) \ @@ -165,14 +165,14 @@ } ## shutdown services - run_command(@("CMD"/L), $targets) + run_command(@("CMD"/L), $primary_target) systemctl stop pe-console-services pe-nginx pxp-agent pe-puppetserver \ pe-orchestration-services puppet pe-puppetdb | CMD # Restore secrets/keys.json if it exists out::message('# Restoring ldap secret key if it exists') - run_command(@("CMD"/L), $targets) + run_command(@("CMD"/L), $primary_target) test -f ${shellquote($recovery_directory)}/rbac/keys.json \ && cp -rp ${shellquote($recovery_directory)}/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ \ || echo secret ldap key doesn\'t exist @@ -181,7 +181,7 @@ # IF restoring orchestrator restore the secrets to /etc/puppetlabs/orchestration-services/conf.d/secrets/ if getvar('recovery_opts.orchestrator') { out::message('# Restoring orchestrator secret keys') - run_command(@("CMD"/L), $targets) + run_command(@("CMD"/L), $primary_target) cp -rp ${shellquote($recovery_directory)}/orchestrator/secrets/* /etc/puppetlabs/orchestration-services/conf.d/secrets/ | CMD } @@ -219,7 +219,7 @@ # Restore database. If there are multiple database restore targets, perform # the restore(s) in parallel. parallelize($database_targets) |$database_target| { - run_command(@("CMD"/L), $targets) + run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/server/bin/pg_restore \ -j 4 \ -d "sslmode=verify-ca \ @@ -261,7 +261,7 @@ # Use `puppet infra` to ensure correct file permissions, restart services, # etc. Make sure not to try and get config data from the classifier, which # isn't yet up and running. - run_command(@("CMD"/L), $targets) + run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/bin/puppet-infrastructure configure --no-recover | CMD @@ -275,7 +275,7 @@ # TODO: consider adding a heuristic to skip when innappropriate due to size # or other factors. if getvar('recovery_opts.puppetdb') and $merge_puppetdb { - run_command(@("CMD"/L), $targets) + run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/bin/puppet-db import \ --cert=$(/opt/puppetlabs/bin/puppet config print hostcert) \ --key=$(/opt/puppetlabs/bin/puppet config print hostprivkey) \ @@ -285,9 +285,9 @@ } # Run Puppet to pick up last remaining config tweaks - run_task('peadm::puppet_runonce', $targets) + run_task('peadm::puppet_runonce', $primary_target) - apply($targets) { + apply($primary_target) { file { $recovery_directory : ensure => 'absent', force => true, diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index 4fbaab71..3c8b8e98 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -6,7 +6,7 @@ $fqdn = run_command('hostname -f', $target) $target.set_var('certname', $fqdn.first['stdout'].chomp) - $command = "echo '${target.vars['certname']} ${target.uri}' | sudo tee -a /etc/hosts" + $command = "echo '${target.uri} ${target.vars['certname']}' | sudo tee -a /etc/hosts" run_command($command, 'localhost') } @@ -14,7 +14,7 @@ # get the latest backup file, if more than one exists $result = run_command('ls -t /tmp/pe-backup*gz | head -1', $primary_host).first.value - $input_file = getvar('result.stdout') + $input_file = strip(getvar('result.stdout')) run_plan('peadm::restore', $primary_host, { 'restore_type' => 'recovery', 'input_file' => $input_file }) From ce15596bae65a1a762e96546084c7a27571cca99 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Thu, 30 Nov 2023 22:38:04 +0100 Subject: [PATCH 047/110] Break by removing classifier database files --- .../test-backup-restore-migration.yaml | 241 ++++++++++++++++++ .github/workflows/test-backup-restore.yaml | 4 +- 2 files changed, 243 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/test-backup-restore-migration.yaml diff --git a/.github/workflows/test-backup-restore-migration.yaml b/.github/workflows/test-backup-restore-migration.yaml new file mode 100644 index 00000000..206844f3 --- /dev/null +++ b/.github/workflows/test-backup-restore-migration.yaml @@ -0,0 +1,241 @@ +--- +name: "Backup and restore test" + +on: + workflow_dispatch: + inputs: + image: + description: "GCP image for test cluster" + required: true + default: "almalinux-cloud/almalinux-8" + architecture: + description: "PE architecture to test" + required: true + default: "standard" + version: + description: "PE version to install" + required: true + default: "2021.7.4" + ssh-debugging: + description: "Boolean; whether or not to pause for ssh debugging" + required: true + default: "false" + +jobs: + backup: + name: "Backup: Cluster A: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" + runs-on: ubuntu-20.04 + env: + BOLT_GEM: true + BOLT_DISABLE_ANALYTICS: true + LANG: "en_US.UTF-8" + + steps: + - name: "Start SSH session" + if: ${{ github.event.inputs.ssh-debugging == 'true' }} + uses: luchihoratiu/debug-via-ssh@main + with: + NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} + SSH_PASS: ${{ secrets.SSH_PASS }} + + - name: "Checkout Source" + uses: actions/checkout@v2 + + - name: "Activate Ruby 2.7" + uses: ruby/setup-ruby@v1 + with: + ruby-version: "2.7" + bundler-cache: true + + - name: "Print bundle environment" + if: ${{ github.repository_owner == 'puppetlabs' }} + run: | + echo ::group::info:bundler + bundle env + echo ::endgroup:: + + - name: "Provision test cluster" + timeout-minutes: 15 + run: | + echo ::group::prepare + mkdir -p $HOME/.ssh + echo 'Host *' > $HOME/.ssh/config + echo ' ServerAliveInterval 150' >> $HOME/.ssh/config + echo ' ServerAliveCountMax 2' >> $HOME/.ssh/config + bundle exec rake spec_prep + echo ::endgroup:: + + echo ::group::provision + bundle exec bolt plan run peadm_spec::provision_test_cluster \ + --modulepath spec/fixtures/modules \ + provider=provision_service \ + image=${{ inputs.image }} \ + architecture=${{ inputs.architecture }} + echo ::endgroup:: + + echo ::group::info:request + cat request.json || true; echo + echo ::endgroup:: + + echo ::group::info:inventory + sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true + echo ::endgroup:: + + # - name: Save inventory file A to an artifact + # uses: actions/upload-artifact@v3 + # with: + # name: inventory_A + # path: spec/fixtures/litmus_inventory.yaml + + - name: "Install PE on test cluster" + timeout-minutes: 120 + run: | + bundle exec bolt plan run peadm_spec::install_test_cluster \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules \ + architecture=${{ inputs.architecture }} \ + version=${{ inputs.version }} + + - name: "Start SSH session" + if: github.event.inputs.ssh-debugging == 'true' + uses: luchihoratiu/debug-via-ssh@main + with: + NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }} + SSH_PASS: ${{ secrets.SSH_PASS }} + + # - name: Download artifacts + # # if: always() + # uses: actions/download-artifact@v3 + # with: + # path: spec/fixtures/ + + - name: perform PE backup of cluster A + timeout-minutes: 10 + continue-on-error: true + run: | + echo ::group::prepare + mkdir -p $HOME/.ssh + echo 'Host *' > $HOME/.ssh/config + echo ' ServerAliveInterval 150' >> $HOME/.ssh/config + echo ' ServerAliveCountMax 2' >> $HOME/.ssh/config + bundle exec rake spec_prep + echo ::endgroup:: + + echo ::group::backup + bundle exec bolt plan run peadm_spec::test_backup \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules + echo ::endgroup:: + + - name: "Wait as long as the file ${HOME}/pause file is present" + continue-on-error: true + # if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} + if: github.event.inputs.ssh-debugging == 'true' + run: | + while [ -f "${HOME}/pause" ] ; do + echo "${HOME}/pause present, sleeping for 60 seconds..." + sleep 10 + done + echo "${HOME}/pause absent, continuing workflow." + + - name: "Tear down cluster A" + if: always() + run: | + if [ -f spec/fixtures/litmus_inventory.yaml ]; then + echo ::group::tear_down + bundle exec rake 'litmus:tear_down' + echo ::endgroup:: + + echo ::group::info:request + cat request.json || true; echo + echo ::endgroup:: + fi + + restore: + name: "Restore: Cluster B: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" + runs-on: ubuntu-20.04 + env: + BOLT_GEM: true + BOLT_DISABLE_ANALYTICS: true + LANG: "en_US.UTF-8" + + steps: + - name: "Checkout Source" + uses: actions/checkout@v2 + + - name: "Activate Ruby 2.7" + uses: ruby/setup-ruby@v1 + with: + ruby-version: "2.7" + bundler-cache: true + + - name: "Print bundle environment" + if: ${{ github.repository_owner == 'puppetlabs' }} + run: | + echo ::group::info:bundler + bundle env + echo ::endgroup:: + + - name: "Provision test cluster" + timeout-minutes: 15 + run: | + echo ::group::prepare + mkdir -p $HOME/.ssh + echo 'Host *' > $HOME/.ssh/config + echo ' ServerAliveInterval 150' >> $HOME/.ssh/config + echo ' ServerAliveCountMax 2' >> $HOME/.ssh/config + bundle exec rake spec_prep + echo ::endgroup:: + + echo ::group::provision + bundle exec bolt plan run peadm_spec::provision_test_cluster \ + --modulepath spec/fixtures/modules \ + provider=provision_service \ + image=${{ inputs.image }} \ + architecture=${{ inputs.architecture }} + echo ::endgroup:: + + echo ::group::info:request + cat request.json || true; echo + echo ::endgroup:: + + echo ::group::info:inventory + sed -e 's/password: .*/password: "[redacted]"/' < spec/fixtures/litmus_inventory.yaml || true + echo ::endgroup:: + + # - name: Save inventory file B to an artifact + # uses: actions/upload-artifact@v3 + # with: + # name: inventory_B + # path: spec/fixtures/litmus_inventory.yaml + + - name: "Install PE on test cluster" + timeout-minutes: 120 + run: | + bundle exec bolt plan run peadm_spec::install_test_cluster \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules \ + architecture=${{ inputs.architecture }} \ + version=${{ inputs.version }} + + - name: Wait for backup to finish + uses: lewagon/wait-on-check-action@v1.3.1 + with: + ref: ${{ github.ref }} + check-name: "Backup: Cluster A: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" + repo-token: ${{ secrets.GITHUB_TOKEN }} + wait-interval: 10 + + - name: "Tear down cluster B" + if: always() + run: | + cp spec/fixtures/inventory_B/litmus_inventory.yaml spec/fixtures/litmus_inventory.yaml || true + if [ -f spec/fixtures/litmus_inventory.yaml ]; then + echo ::group::tear_down + bundle exec rake 'litmus:tear_down' + echo ::endgroup:: + + echo ::group::info:request + cat request.json || true; echo + echo ::endgroup:: + fi diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 914205b3..12b0fa6c 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -119,8 +119,8 @@ jobs: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) echo "Removing ssl directories" bundle exec bolt command run "rm -rf /etc/puppetlabs/puppetserver/ca /etc/puppetlabs/puppet/ssl" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - echo "Removing config file" - bundle exec bolt command run "rm /etc/puppetlabs/enterprise/conf.d/pe.conf" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + echo "Removing classifier database" + bundle exec bolt command run "rm -rf /opt/puppetlabs/server/data/postgresql/classifier" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml From 55131f1d545eb658c774ed4855de8551fc08fee6 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 1 Dec 2023 13:13:56 +0100 Subject: [PATCH 048/110] add plan to edit inventory file --- .../peadm_spec/plans/add_inventory_hostnames.pp | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 spec/acceptance/peadm_spec/plans/add_inventory_hostnames.pp diff --git a/spec/acceptance/peadm_spec/plans/add_inventory_hostnames.pp b/spec/acceptance/peadm_spec/plans/add_inventory_hostnames.pp new file mode 100644 index 00000000..89914ac6 --- /dev/null +++ b/spec/acceptance/peadm_spec/plans/add_inventory_hostnames.pp @@ -0,0 +1,13 @@ +plan peadm_spec::add_inventory_hostnames( + String[1] $inventory_file +) { + $t = get_targets('*') + wait_until_available($t) + + parallelize($t) |$target| { + $fqdn = run_command('hostname -f', $target) + $target.set_var('certname', $fqdn.first['stdout'].chomp) + $command = "yq eval '(.groups[].targets[] | select(.uri == \"${target.uri}\").name) = \"${target.vars['certname']}\"' -i ${inventory_file}" + run_command($command, 'localhost') + } +} From 710c2d84651c4aa5eef679472bf8b17f7401ec1f Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 1 Dec 2023 13:15:19 +0100 Subject: [PATCH 049/110] merge recovery backup and restore --- plans/backup.pp | 55 +++++++++++++++++++++----------- plans/restore.pp | 81 +++++++++++++++++++++++++++--------------------- 2 files changed, 82 insertions(+), 54 deletions(-) diff --git a/plans/backup.pp b/plans/backup.pp index e38ce48f..42b821c1 100644 --- a/plans/backup.pp +++ b/plans/backup.pp @@ -89,6 +89,16 @@ mode => '0711', } } + + if $backup_type == 'recovery' { + # create a backup subdir for recovery configuration + file { "${backup_directory}/recovery": + ensure => 'directory', + owner => 'root', + group => 'root', + mode => '0711', + } + } } if getvar('recovery_opts.classifier') { @@ -98,30 +108,39 @@ ) } - if getvar('recovery_opts.ca') { - out::message('# Backing up ca and ssl certificates') + if $backup_type == 'recovery' { + out::message('# Backing up ca, certs, code and config for recovery') # lint:ignore:strict_indent run_command(@("CMD"), $targets) - /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/ca --scope=certs + /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/recovery \ + --scope=certs,code,config | CMD - } +# lint:endignore + } else { + if getvar('recovery_opts.ca') { + out::message('# Backing up ca and ssl certificates') + # lint:ignore:strict_indent + run_command(@("CMD"), $targets) + /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/ca --scope=certs + | CMD + } - if getvar('recovery_opts.code') { - out::message('# Backing up code') - # run_command("chown pe-postgres ${shellquote($backup_directory)}/code", $targets) - run_command(@("CMD"), $targets) - /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/code --scope=code - | CMD - } + if getvar('recovery_opts.code') { + out::message('# Backing up code') + # run_command("chown pe-postgres ${shellquote($backup_directory)}/code", $targets) + run_command(@("CMD"), $targets) + /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/code --scope=code + | CMD + } - if getvar('recovery_opts.config') { - out::message('# Backing up config') - run_command("chown pe-postgres ${shellquote($backup_directory)}/config", $targets) - run_command(@("CMD"), $targets) - /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/config --scope=config - | CMD + if getvar('recovery_opts.config') { + out::message('# Backing up config') + run_command("chown pe-postgres ${shellquote($backup_directory)}/config", $targets) + run_command(@("CMD"), $targets) + /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/config --scope=config + | CMD + } } - # Check if /etc/puppetlabs/console-services/conf.d/secrets/keys.json exists and if so back it up if getvar('recovery_opts.rbac') { out::message('# Backing up ldap secret key if it exists') diff --git a/plans/restore.pp b/plans/restore.pp index 9dc1a3ed..e83bf064 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -18,9 +18,6 @@ # Path to the recovery tarball Pattern[/.*\.tar\.gz$/] $input_file, - - # Do we want to merge existing puppetdb content to the restored content? - Boolean $merge_puppetdb = false, ) { peadm::assert_supported_bolt_version() @@ -116,45 +113,57 @@ ) } - if getvar('recovery_opts.ca') { - out::message('# Restoring ca and ssl certificates') -# lint:ignore:strict_indent - run_command(@("CMD"/L), $primary_target) - /opt/puppetlabs/bin/puppet-backup restore \ - --scope=certs \ - --tempdir=${shellquote($recovery_directory)} \ - --force \ - ${shellquote($recovery_directory)}/ca/pe_backup-*tgz - | CMD - } + if $restore_type == 'recovery' { + out::message('# Restoring ca, certs, code and config for recovery') + # lint:ignore:strict_indent + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/bin/puppet-backup restore \ + --scope=certs,code,config \ + --tempdir=${shellquote($recovery_directory)} \ + --force \ + ${shellquote($recovery_directory)}/recovery/pe_backup-*tgz + | CMD + # lint:endignore + } else { + if getvar('recovery_opts.ca') { + out::message('# Restoring ca and ssl certificates') + # lint:ignore:strict_indent + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/bin/puppet-backup restore \ + --scope=certs \ + --tempdir=${shellquote($recovery_directory)} \ + --force \ + ${shellquote($recovery_directory)}/ca/pe_backup-*tgz + | CMD + } - if getvar('recovery_opts.code') { - out::message('# Restoring code') - run_command(@("CMD"/L), $primary_target) - /opt/puppetlabs/bin/puppet-backup restore \ - --scope=code \ - --tempdir=${shellquote($recovery_directory)} \ - --force \ - ${shellquote($recovery_directory)}/code/pe_backup-*tgz - | CMD - } + if getvar('recovery_opts.code') { + out::message('# Restoring code') + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/bin/puppet-backup restore \ + --scope=code \ + --tempdir=${shellquote($recovery_directory)} \ + --force \ + ${shellquote($recovery_directory)}/code/pe_backup-*tgz + | CMD + } - if getvar('recovery_opts.config') { - out::message('# Restoring config') - run_command(@("CMD"/L), $primary_target) - /opt/puppetlabs/bin/puppet-backup restore \ - --scope=config \ - --tempdir=${shellquote($recovery_directory)} \ - --force \ - ${shellquote($recovery_directory)}/config/pe_backup-*tgz - | CMD + if getvar('recovery_opts.config') { + out::message('# Restoring config') + run_command(@("CMD"/L), $primary_target) + /opt/puppetlabs/bin/puppet-backup restore \ + --scope=config \ + --tempdir=${shellquote($recovery_directory)} \ + --force \ + ${shellquote($recovery_directory)}/config/pe_backup-*tgz + | CMD + } } - # Use PuppetDB's /pdb/admin/v1/archive API to SAVE data currently in PuppetDB. # Otherwise we'll completely lose it if/when we restore. # TODO: consider adding a heuristic to skip when innappropriate due to size # or other factors. - if getvar('recovery_opts.puppetdb') and $merge_puppetdb { + if getvar('recovery_opts.puppetdb') and $restore_type == 'migration' { out::message('# Exporting puppetdb') run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/bin/puppet-db export \ @@ -274,7 +283,7 @@ # into the restored database. # TODO: consider adding a heuristic to skip when innappropriate due to size # or other factors. - if getvar('recovery_opts.puppetdb') and $merge_puppetdb { + if getvar('recovery_opts.puppetdb') and $restore_type == 'migration' { run_command(@("CMD"/L), $primary_target) /opt/puppetlabs/bin/puppet-db import \ --cert=$(/opt/puppetlabs/bin/puppet config print hostcert) \ From bf9878600079ce762d65067db89358424cd4b370 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 1 Dec 2023 14:58:18 +0100 Subject: [PATCH 050/110] Modify inventory to include name Co-authored-by: Neil Anderson --- .../peadm_spec/plans/test_restore.pp | 21 ++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index 3c8b8e98..c42dc46b 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -4,13 +4,20 @@ parallelize($t) |$target| { $fqdn = run_command('hostname -f', $target) - $target.set_var('certname', $fqdn.first['stdout'].chomp) + $certname = $fqdn.first['stdout'].chomp + $target.set_var('certname', $certname) + } - $command = "echo '${target.uri} ${target.vars['certname']}' | sudo tee -a /etc/hosts" - run_command($command, 'localhost') + $targets_with_name = $t.map |$target| { + Target.new({ + 'uri' => $target.uri, + 'name' => $target.vars['certname'], + 'config' => $target.config, + 'vars' => $target.vars, + }) } - $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] + $primary_host = $targets_with_name.filter |$n| { $n.vars['role'] == 'primary' }[0] # get the latest backup file, if more than one exists $result = run_command('ls -t /tmp/pe-backup*gz | head -1', $primary_host).first.value @@ -20,11 +27,11 @@ # run infra status on the primary out::message("Running peadm::status on primary host ${primary_host}") - $result = run_plan('peadm::status', $primary_host, { 'format' => 'json' }) + $status = run_plan('peadm::status', $primary_host, { 'format' => 'json' }) - out::message($result) + out::message($status) - if empty($result['failed']) { + if empty($status['failed']) { out::message('Cluster is healthy, continuing') } else { fail_plan('Cluster is not healthy, aborting') From 3b7cedde2f70c16bf9117c2ef327f5ea07a4eeca Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 1 Dec 2023 15:41:00 +0100 Subject: [PATCH 051/110] stop breaking classifier db --- .github/workflows/test-backup-restore.yaml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 12b0fa6c..241444e1 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -106,7 +106,8 @@ jobs: echo ::group::backup bundle exec bolt plan run peadm_spec::test_backup \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ - --modulepath spec/fixtures/modules + --modulepath spec/fixtures/modules \ + --stream echo ::endgroup:: - name: Set up yq @@ -119,8 +120,8 @@ jobs: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) echo "Removing ssl directories" bundle exec bolt command run "rm -rf /etc/puppetlabs/puppetserver/ca /etc/puppetlabs/puppet/ssl" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - echo "Removing classifier database" - bundle exec bolt command run "rm -rf /opt/puppetlabs/server/data/postgresql/classifier" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + # echo "Removing classifier database" + # bundle exec bolt command run "rm -rf /opt/puppetlabs/server/data/postgresql/classifier" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml From 7355630562fc820d21aed627fae333d5462e8978 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sat, 2 Dec 2023 17:34:35 +0100 Subject: [PATCH 052/110] add PE reinstall step --- .github/workflows/test-backup-restore.yaml | 7 ++++++- .../acceptance/peadm_spec/tasks/reinstall_pe.json | 14 ++++++++++++++ spec/acceptance/peadm_spec/tasks/reinstall_pe.sh | 15 +++++++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-) create mode 100644 spec/acceptance/peadm_spec/tasks/reinstall_pe.json create mode 100644 spec/acceptance/peadm_spec/tasks/reinstall_pe.sh diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 241444e1..e872dbc2 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -125,8 +125,13 @@ jobs: bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + - name: Reinstall PE primary in a non-peadm way + run: | + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt task run reinstall_pe version=${{ inputs.version }} -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + - name: Perform peadm restore of cluster - timeout-minutes: 10 + timeout-minutes: 30 continue-on-error: true run: | echo ::group::prepare diff --git a/spec/acceptance/peadm_spec/tasks/reinstall_pe.json b/spec/acceptance/peadm_spec/tasks/reinstall_pe.json new file mode 100644 index 00000000..a9668514 --- /dev/null +++ b/spec/acceptance/peadm_spec/tasks/reinstall_pe.json @@ -0,0 +1,14 @@ +{ + "description": "Delete a certname from the Postgres DB", + "parameters": { + "version": { + "type": "String[1]", + "description": "The PE version to install" + }, + "arch": { + "type": "String[1]", + "description": "The hardware architecture to install", + "default": "el-8-x86_64" + } + } +} diff --git a/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh b/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh new file mode 100644 index 00000000..7c799a84 --- /dev/null +++ b/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh @@ -0,0 +1,15 @@ +#!/bin/bash -e +# This task reinstalls PE and needs to run as root. +# It assumes el-8-x86_64 + +# Uninstall PE if installed +/opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d -y || true + +# Download PE +INSTALLER="puppet-enterprise-${PT_version}-${PT_arch}" +curl -O "https://s3.amazonaws.com/pe-builds/released/${PT_version}/${INSTALLER}.tar.gz" +tar xf "${INSTALLER}.tar.gz" + +# Install PE. We need to pass "y" through stdin since the flag -y requires pe.conf to be present. +cd $INSTALLER +echo 'y' | ./puppet-enterprise-installer \ No newline at end of file From a7efdc7bd56d4bedaf55df27608f54c16b5f65b8 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sat, 2 Dec 2023 17:36:19 +0100 Subject: [PATCH 053/110] reinstate remove classifier db --- .github/workflows/test-backup-restore.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index e872dbc2..17d4619e 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -120,8 +120,8 @@ jobs: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) echo "Removing ssl directories" bundle exec bolt command run "rm -rf /etc/puppetlabs/puppetserver/ca /etc/puppetlabs/puppet/ssl" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - # echo "Removing classifier database" - # bundle exec bolt command run "rm -rf /opt/puppetlabs/server/data/postgresql/classifier" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + echo "Removing classifier database" + bundle exec bolt command run "rm -rf /opt/puppetlabs/server/data/postgresql/classifier" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml From 0a54f833b222426a512b52f5402511356faace50 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sat, 2 Dec 2023 18:54:17 +0100 Subject: [PATCH 054/110] Fix calling the reinstall task --- .github/workflows/test-backup-restore.yaml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 17d4619e..947c8a99 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -128,7 +128,10 @@ jobs: - name: Reinstall PE primary in a non-peadm way run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt task run reinstall_pe version=${{ inputs.version }} -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + bundle exec bolt task run peadm_spec::reinstall_pe version=${{ inputs.version }} -t $primary \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules + --verbose --stream - name: Perform peadm restore of cluster timeout-minutes: 30 From 694561fb7d8fd96bbdf69123e9b5794637599e7e Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sat, 2 Dec 2023 21:09:36 +0100 Subject: [PATCH 055/110] Add continue on error for when reinstall exits with non-zero code --- .github/workflows/test-backup-restore.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 947c8a99..1ae25154 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -126,6 +126,7 @@ jobs: bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - name: Reinstall PE primary in a non-peadm way + continue-on-error: true run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt task run peadm_spec::reinstall_pe version=${{ inputs.version }} -t $primary \ From 074a69dbcf6196fd5258db8962475fb41dfe9c1c Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sun, 3 Dec 2023 02:15:49 +0100 Subject: [PATCH 056/110] some textual fixes --- spec/acceptance/peadm_spec/tasks/reinstall_pe.json | 4 ++-- spec/acceptance/peadm_spec/tasks/reinstall_pe.sh | 1 - 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/spec/acceptance/peadm_spec/tasks/reinstall_pe.json b/spec/acceptance/peadm_spec/tasks/reinstall_pe.json index a9668514..fb2ccd33 100644 --- a/spec/acceptance/peadm_spec/tasks/reinstall_pe.json +++ b/spec/acceptance/peadm_spec/tasks/reinstall_pe.json @@ -1,5 +1,5 @@ { - "description": "Delete a certname from the Postgres DB", + "description": "Reinstall PE", "parameters": { "version": { "type": "String[1]", @@ -7,7 +7,7 @@ }, "arch": { "type": "String[1]", - "description": "The hardware architecture to install", + "description": "The PE installation platform", "default": "el-8-x86_64" } } diff --git a/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh b/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh index 7c799a84..5175a63e 100644 --- a/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh +++ b/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh @@ -1,6 +1,5 @@ #!/bin/bash -e # This task reinstalls PE and needs to run as root. -# It assumes el-8-x86_64 # Uninstall PE if installed /opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d -y || true From dc0c3763a5796a4c9e18db7dfd16c339b000c2ef Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 4 Dec 2023 15:52:01 +0100 Subject: [PATCH 057/110] update docs --- documentation/backup.md | 74 ---------------------- documentation/backup_restore.md | 108 ++++++++++++++++++++++++++++++++ documentation/restore.md | 46 -------------- 3 files changed, 108 insertions(+), 120 deletions(-) delete mode 100644 documentation/backup.md create mode 100644 documentation/backup_restore.md delete mode 100644 documentation/restore.md diff --git a/documentation/backup.md b/documentation/backup.md deleted file mode 100644 index 5e19daa3..00000000 --- a/documentation/backup.md +++ /dev/null @@ -1,74 +0,0 @@ -# Backup Puppet Enterprise using the PEADM module - -## What is being backed up? - -By default the `peadm::backup` plan will backup the following items: - -1. Orchestrator -2. PuppetDB -3. RBAC - - Also, It copies the LDAP secret key if it present -4. Activity -5. Classification - -Optionally you can also backup: - -1. CA (CA and SSL certificates) - ----- - -Most of the backups will be a direct copy of the databases with the exception of: - -- Classification - - The backup is done using an API call -- CA - - The certificate files will be copy via the puppet-backup script. - - -**Note:** - -It is important to highlight that the `peadm::backup` plan's output is different than the one you will get when you backup manually using [the `puppet-backup create` command.](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure). - -The differences between these two backup tools are: - -1. The structure of the backup file, since this plan `peadm:backup` uses a combination of scripts, API calls, and DB backups, you will not be able to restore it using the traditional `sudo puppet-backup restore ` command. [To read more about the difference between the options (flags), please read the official PE doc for backing up & restoring.](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure) - -1. The core idea of the `peadm:backup` plan is a focus on being able to separate customers data from the infrastructure data. This contrasts with the PE backup script which is more focused on a traditional backup and restore after a system failure where will be restoring the same infrastructure. It, therefore, targets things like backing up PostgreSQL rather than the actual data you want. - - This can be seen when restoring node groups with `peadm:backup` only restores the non-PEinfrastructure source groups and restores them to target. - - - [PE backup](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure) is not capable of restoring to a DR setup and requires you to provision a new DR setup whereas [`peadm:restore`](restore.md) will reinitialise replicas as part of the process - -1. One of the other key differences with the `peadm:backup` is it is compatible with the extra-large (XL) architectures as it can communicate with the remote PostgreSQL server using temporary escalated privileges. - -## How can I customize my backup? - -We need to pass the `backup` parameter to the `peadm::backup` plan. - -**Example** - -**Note:** The `peadm::backup` plan can only be executed from the PE primary server. - -Let's Backup _only_ RBAC - -``` -# backup_params.json - -{ - "backup": { - "orchestrator": false, - "puppetdb": false, - "rbac": true, - "activity": false, - "ca": false, - "classifier": false - }, - "output_directory": "/tmp" -} -``` - -We selected our backup options and the `output_directory` (default `/tmp`). - -To run the backup plan with our custom parameters: - - bolt plan run peadm::backup --params @backup_params.json - diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md new file mode 100644 index 00000000..073d46d1 --- /dev/null +++ b/documentation/backup_restore.md @@ -0,0 +1,108 @@ +# Backup and Restore Puppet Enterprise using the PEADM module + +## Introduction + +The PEADM backup and restore supports two types: `recovery` and `custom`. +You can select the type by providing the parameter `backup_type` for backup and `restore_type` for restore. +`recovery` is the default backup/restore type. + +The backup file will be named `pe-backup-YYYY-MM-DDTHHMMSSZ.tar.gz` and placed in `/tmp` by default. The output directory can be overridden with +the parameter `output_directory`. + +The restore needs to specify the backup file path to restore from in its `input_file` parameter. + +**Important note**: peadm backup and restore can only be used on PE installs which are created using PEADM. Also, you can only use `peadm::restore` on a backup created by `peadm::backup`. + +A backup and restore need to specify the primary server as the target. + +## Recovery backup type + +When backup type `recovery` is selected, the primary backup created is intended to be used to recover the primary it if it fails. This means that the primary's configuration will be restored to exactly the state it was in when the backup was created. + +You can create a recovery backup as follows: +``` +bolt plan run peadm::backup --targets my.primary.vm backup_type=recovery +``` +or, simply, +``` +bolt plan run peadm::backup --targets my.primary.vm +``` + +To restore from this backup, you can do: +``` +bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" +``` + +The recovery restore will also work if some or all services are down on the primary. The restore process will restart the services at the end. + +## Custom backup type + +When selecting the `custom` backup type, the user can choose which items are backed up and/or restored using the `backup` (or `restore`) plan parameter. +This parameter can be omitted, in which case all options will be turned on by default. + +To specify custom backup (restore) options, it is best to create a `params.json` file like follows: + +```json +{ + "backup_type" : "custom", + "backup": { + "activity" : false, + "ca" : true, + "classifier" : false, + "code" : true, + "config" : true, + "orchestrator" : false, + "puppetdb" : true, + "rbac" : false + } +} +``` + +To backup using this parameter file, do: +``` +bolt plan run peadm::backup --targets my.primary.vm --params @params.json +``` + +or, for restore, + +```json +{ + "restore_type" : "custom", + "restore": { + "activity" : false, + "ca" : true, + "classifier" : false, + "code" : true, + "config" : true, + "orchestrator" : false, + "puppetdb" : true, + "rbac" : false, + }, + "input_file" : "/tmp/my_backup.tar.gz" +} +``` + +To restore using this parameter file, do: +``` +bolt plan run peadm::restore --targets my.primary.vm --params @params.json +``` + +## What exactly is backed up and restored? + +These are the explanations of the different backup/restore items: + +| Item | Comments | Used in `recovery` | +| ------------ | ------------------------------------------------------------------------------------------------------ | ------------------ | +| activity | Activity database | | +| ca | CA and ssl certificates | ✅ | +| classifier | Classifier database. Restore will merge user-defined node groups and not overwrite system node groups. | | +| code | Code directory | ✅ | +| config | Configuration files and databases (databases are restored literally) | ✅ | +| orchestrator | Orchestrator database | | +| puppetdb | PuppetDB database (including support for XL where puppetdb is running on an external db server) | ✅ | +| rbac | RBAC database | | + +**Note**: `ca`, `code` and `config` are backed up using the `puppet-backup create` command and restored using the `puppet-backup restore` command. +The `config` item includes backups of `activity`, `classifier`, `orchestrator` and `rbac` databases. + +**Note:** It is important to highlight that the `peadm::backup` plan's output is different than the one you will get when you backup manually using [the `puppet-backup create` command.](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure). \ No newline at end of file diff --git a/documentation/restore.md b/documentation/restore.md deleted file mode 100644 index 50331ead..00000000 --- a/documentation/restore.md +++ /dev/null @@ -1,46 +0,0 @@ -# Restore Puppet Enterprise using the PEADM module - -Once you have a [backup](backup.md) you can restore a PE primary server. It is important to highlight that you can not use the `peadm::restore` plan with a backup that was not created with the `peadm::backup` plan. - -**Things to consider** - -There is some downtime involved when the `peadm:restore` plan is executed. The following services will be restarted: - -1. pe-console-services -2. pe-nginx -3. pxp-agent -4. pe-puppetserver -5. pe-orchestration-services -6. puppet -7. pe-puppetdb - -There is also a procedure related to the restoration of the databases where `peadm::restore` will temporarily set privileges (permissions) to a DB user with the only purpose of restoring the database. These temporary privileges are removed right after the PostgreSQL restore command finishes. - -Also, this plan uses internal calls to the `peadm::get_targets` function, this means the plan expects the services to be up and running when you start either a restore or a backup. - -## How to use the restore plan? - -As in the `peadm::backup` plan, you can choose what you want to restore by specifying the parameter `restore`. The `input_file` parameter refers to the location of the backup tarball. - -Example: - -**Note:** The `peadm::restore` plan can only be executed from the PE primary server. - -``` -# restore_params.json - -{ - "restore": { - "orchestrator": false, - "puppetdb": false, - "rbac": true, - "activity": false, - "ca": false, - "classifier": false, - }, - "input_file": "/tmp/path/backup_tarball.tgz" -} -``` -To run the `peadm::restore` plan with our custom parameters file, we can do: - - bolt plan run peadm::restore --params @restore_params.json From 7a99564e6e9eedcb0d62b66fe17b5d086e08a846 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 4 Dec 2023 15:56:35 +0100 Subject: [PATCH 058/110] add in-code docs --- plans/backup.pp | 20 ++++++++++++-------- plans/restore.pp | 17 +++++++++-------- 2 files changed, 21 insertions(+), 16 deletions(-) diff --git a/plans/backup.pp b/plans/backup.pp index 42b821c1..493fab35 100644 --- a/plans/backup.pp +++ b/plans/backup.pp @@ -1,20 +1,24 @@ -# @api private -# @summary Backup the core user settings for puppet infrastructure +# @summary Backup puppet primary configuration +# +# @param targets These are a list of the primary puppetservers from one or multiple puppet clusters +# @param backup_type Currently, the recovery and custom backup types are supported +# @param backup A hash of custom backup options, see the peadm::recovery_opts_default() function for the default values +# @param output_directory The directory to place the backup in +# @example +# bolt plan run peadm::backup -t primary1.example.com,primary2.example.com # -# This plan can backup data as outlined at insert doc -# plan peadm::backup ( # This plan should be run on the primary server - Peadm::SingleTargetSpec $targets, + Peadm::SingleTargetSpec $targets, # backup type determines the backup options - Enum['recovery', 'migration', 'custom'] $backup_type = 'recovery', + Enum['recovery', 'custom'] $backup_type = 'recovery', # Which data to backup - Peadm::Recovery_opts $backup = {}, + Peadm::Recovery_opts $backup = {}, # Where to put the backup folder - String $output_directory = '/tmp', + String $output_directory = '/tmp', ) { peadm::assert_supported_bolt_version() diff --git a/plans/restore.pp b/plans/restore.pp index e83bf064..7fba2a12 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -1,17 +1,18 @@ -# @api private -# @summary Restore the core user settings for puppet infrastructure from backup +# @summary Restore puppet primary configuration +# +# @param targets These are a list of the primary puppetservers from one or multiple puppet clusters +# @param backup_type Currently, the `recovery` and `custom` restore types are supported +# @param restore A hash of custom backup options, see the peadm::recovery_opts_default() function for the default values +# @param input_file The file containing the backup to restore from +# @example +# bolt plan run peadm::restore -t primary1.example.com,primary2.example.com input_file=/tmp/peadm-backup.tar.gz # -# This plan can restore data to puppet infrastructure for DR and rebuilds -# -# TODO -# - make sure restore tries to leave the system in a running state if possible -# - potentially merge the restore of certs, config and code scope to one puppet-backup restore command plan peadm::restore ( # This plan should be run on the primary server Peadm::SingleTargetSpec $targets, # restore type determines the restore options - Enum['recovery', 'migration', 'custom'] $restore_type = 'recovery', + Enum['recovery', 'custom'] $restore_type = 'recovery', # Which data to restore Peadm::Recovery_opts $restore = {}, From 11673d64de03f35faf4998db50f090d02dd8b524 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Tue, 5 Dec 2023 10:49:48 +0000 Subject: [PATCH 059/110] Adding PR review trigger --- .github/workflows/test-backup-restore.yaml | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 1ae25154..774df817 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -2,6 +2,9 @@ name: "Backup and restore test" on: + pull_request_target: + types: [ready_for_review] + workflow_dispatch: inputs: image: @@ -23,7 +26,8 @@ on: jobs: backup: - name: "Backup cluster: PE ${{ inputs.version }} ${{ inputs.architecture }} on ${{ inputs.image }}" + name: "Backup cluster: PE ${{ github.event.inputs.version || '2023.5.0' }} ${{ github.event.inputs.architecture || 'extra-large' }} on ${{ github.event.inputs.image || 'almalinux-cloud/almalinux-8' }}" + runs-on: ubuntu-20.04 env: BOLT_GEM: true @@ -69,8 +73,8 @@ jobs: bundle exec bolt plan run peadm_spec::provision_test_cluster \ --modulepath spec/fixtures/modules \ provider=provision_service \ - image=${{ inputs.image }} \ - architecture=${{ inputs.architecture }} + image=${{ github.event.inputs.image || 'almalinux-cloud/almalinux-8' }} \ + architecture=${{ github.event.inputs.architecture || 'extra-large' }} echo ::endgroup:: echo ::group::info:request @@ -87,8 +91,8 @@ jobs: bundle exec bolt plan run peadm_spec::install_test_cluster \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ - architecture=${{ inputs.architecture }} \ - version=${{ inputs.version }} \ + architecture=${{ github.event.inputs.architecture || 'extra-large' }} \ + version=${{ github.event.inputs.version || '2023.5.0' }} \ --stream - name: Perform peadm backup of cluster @@ -129,7 +133,7 @@ jobs: continue-on-error: true run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt task run peadm_spec::reinstall_pe version=${{ inputs.version }} -t $primary \ + bundle exec bolt task run peadm_spec::reinstall_pe version=${{ github.event.inputs.version || '2023.5.0' }} -t $primary \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules --verbose --stream From cc0bdc1a03211dae6fe633c536a080e5b0ca633e Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Wed, 6 Dec 2023 19:23:26 +0100 Subject: [PATCH 060/110] Add TODOs --- .github/workflows/test-backup-restore.yaml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 774df817..f22fa6ab 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -1,4 +1,8 @@ --- +# TODO: +# - [ ] test on XL, also break the puppetdb database +# - [ ] test custom scenario for only restoring primary +# - [ ] test custom scenario for only restoring puppetdb (on external server) name: "Backup and restore test" on: From c5163b7c881ce85a724ed4940ec6aa7210ab154b Mon Sep 17 00:00:00 2001 From: JHunniford <97452751+J-Hunniford@users.noreply.github.com> Date: Thu, 7 Dec 2023 12:31:37 +0000 Subject: [PATCH 061/110] Edit backup_restore (first pass) --- documentation/backup_restore.md | 85 +++++++++++++++++++-------------- 1 file changed, 48 insertions(+), 37 deletions(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 073d46d1..ae90b5d3 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -1,47 +1,59 @@ -# Backup and Restore Puppet Enterprise using the PEADM module +# Backup and restore Puppet Enterprise (PE) -## Introduction +If your PE installation is managed by `peadm`, you can back up and restore PE using this process: +1. Use the `peadm::backup` plan to create a backup of your primary server. +2. Use the `peadm::restore` plan to restore PE from a `peadm::backup`. -The PEADM backup and restore supports two types: `recovery` and `custom`. -You can select the type by providing the parameter `backup_type` for backup and `restore_type` for restore. -`recovery` is the default backup/restore type. +**Important:** If your PE installation is not managed by `peadm`, you cannot use the `peadm::backup` and `peadm::restore` plans. For information on converting to a `peadm` managed installation, see [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md). -The backup file will be named `pe-backup-YYYY-MM-DDTHHMMSSZ.tar.gz` and placed in `/tmp` by default. The output directory can be overridden with -the parameter `output_directory`. +When running the backup and restore plans, you can define the `backup_type` and `restore_type` parameters with either of the following values: +* `recovery`: Use this type to create a full backup of your primary server, including data for all services. This allows you to restore your PE installation to the exact state it was in at the time of the backup. +* `custom`: Use this type when you want to selectively back up and restore data for specific services. -The restore needs to specify the backup file path to restore from in its `input_file` parameter. +If no type is specified, the default is `recovery`. -**Important note**: peadm backup and restore can only be used on PE installs which are created using PEADM. Also, you can only use `peadm::restore` on a backup created by `peadm::backup`. +When backing up or restoring PE, you must use the `--targets` option to specify the hostname (FQDN) of your primary server. + +The backup file created is named `pe-backup-YYYY-MM-DDTHHMMSSZ.tar.gz` and placed by default in `/tmp`. To specify a different location for the backup file, you can define the `output_directory` parameter. -A backup and restore need to specify the primary server as the target. +This example shows how to run a `recovery` backup which places the backup file in a custom location. +``` +bolt plan run peadm::backup --targets my.primary.vm backup_type=recovery output_directory=/custom_path +``` + +When restoring PE, you must define the `input_file` parameter to specify the path to the backup file you want to use. For example: +``` +bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" +``` -## Recovery backup type +## Using `recovery` backup and restore -When backup type `recovery` is selected, the primary backup created is intended to be used to recover the primary it if it fails. This means that the primary's configuration will be restored to exactly the state it was in when the backup was created. +When you run a `recovery` backup plan, the primary server configuration is backed up in full. In the event of a primary server failure, this backup can be used to restore the PE installation to the state it was in when the backup was created. -You can create a recovery backup as follows: +You can create a `recovery` backup as follows: ``` bolt plan run peadm::backup --targets my.primary.vm backup_type=recovery ``` -or, simply, +Alternatively, because `recovery` is the default type, you can use this simplified command: ``` bolt plan run peadm::backup --targets my.primary.vm ``` -To restore from this backup, you can do: +To restore your installation from this backup, run: ``` bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" ``` -The recovery restore will also work if some or all services are down on the primary. The restore process will restart the services at the end. +**Tip**: Restoring from a `recovery` backup restarts any services that are unavailable on the primary server. -## Custom backup type +## Using `custom` backup and restore -When selecting the `custom` backup type, the user can choose which items are backed up and/or restored using the `backup` (or `restore`) plan parameter. -This parameter can be omitted, in which case all options will be turned on by default. +To specify the items that are backed up and restored, define the `backup_type` or `restore_type` parameters as `custom`. +Otherwise, the default type is `recovery`. -To specify custom backup (restore) options, it is best to create a `params.json` file like follows: +To specify the `custom` items, you can create and reference `params.json` files as shown in the following examples. +To specify custom backup options: ```json { "backup_type" : "custom", @@ -58,12 +70,12 @@ To specify custom backup (restore) options, it is best to create a `params.json` } ``` -To backup using this parameter file, do: +To create a backup using the options specified in this parameter file, run: ``` bolt plan run peadm::backup --targets my.primary.vm --params @params.json ``` -or, for restore, +To specify custom restore options: ```json { @@ -82,27 +94,26 @@ or, for restore, } ``` -To restore using this parameter file, do: +To restore PE using the options specified in this parameter file, run: ``` bolt plan run peadm::restore --targets my.primary.vm --params @params.json ``` ## What exactly is backed up and restored? -These are the explanations of the different backup/restore items: +The following table shows the items you can specify and indicates what is included in `recovery`: -| Item | Comments | Used in `recovery` | -| ------------ | ------------------------------------------------------------------------------------------------------ | ------------------ | -| activity | Activity database | | -| ca | CA and ssl certificates | ✅ | -| classifier | Classifier database. Restore will merge user-defined node groups and not overwrite system node groups. | | -| code | Code directory | ✅ | -| config | Configuration files and databases (databases are restored literally) | ✅ | -| orchestrator | Orchestrator database | | -| puppetdb | PuppetDB database (including support for XL where puppetdb is running on an external db server) | ✅ | -| rbac | RBAC database | | +| Data or service | Explanation | Used in `recovery` | +| ------------------| -------------------------------------------------------------------------------------------------------- | ------------------ | +| `activity ` | Activity database | | +| `ca ` | CA and ssl certificates | ✅ | +| `classifier` | Classifier database. Restore merges user-defined node groups rather than overwriting system node groups. | | +| `code` | Code directory | ✅ | +| `config` | Configuration files and databases (databases are restored literally) | ✅ | +| `orchestrator ` | Orchestrator database | | +| `puppetdb` | PuppetDB database (including support for XL where puppetdb is running on an external db server) | ✅ | +| `rbac` | RBAC database | | -**Note**: `ca`, `code` and `config` are backed up using the `puppet-backup create` command and restored using the `puppet-backup restore` command. -The `config` item includes backups of `activity`, `classifier`, `orchestrator` and `rbac` databases. +**Note**: The `peadm` backup and restore plans utilize the `puppet-backup` tool for backing up and restoring `ca`, `code` and `config`. For `config`, the data backed up includes the `activity`, `classifier`, `orchestrator`, and `rbac` databases. -**Note:** It is important to highlight that the `peadm::backup` plan's output is different than the one you will get when you backup manually using [the `puppet-backup create` command.](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure). \ No newline at end of file +**Note:** The output for the `peadm::backup` plan differs from the output that is returned when you manually run the [`puppet-backup create` command](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure). From ce347a01456bab580f8920c48bb284f6b17b4532 Mon Sep 17 00:00:00 2001 From: JHunniford <97452751+J-Hunniford@users.noreply.github.com> Date: Thu, 7 Dec 2023 14:59:47 +0000 Subject: [PATCH 062/110] Edit to be explicit about primary and db --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index ae90b5d3..77a8fc65 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -7,7 +7,7 @@ If your PE installation is managed by `peadm`, you can back up and restore PE us **Important:** If your PE installation is not managed by `peadm`, you cannot use the `peadm::backup` and `peadm::restore` plans. For information on converting to a `peadm` managed installation, see [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md). When running the backup and restore plans, you can define the `backup_type` and `restore_type` parameters with either of the following values: -* `recovery`: Use this type to create a full backup of your primary server, including data for all services. This allows you to restore your PE installation to the exact state it was in at the time of the backup. +* `recovery`: Use this type to create a full backup of your primary server, including data for all services. This allows you to restore your primary server and all services (including database services running on external servers) to the exact state they were in at the time of the backup. * `custom`: Use this type when you want to selectively back up and restore data for specific services. If no type is specified, the default is `recovery`. From 3435e413f3564f060af5f76006c3b8335902ec96 Mon Sep 17 00:00:00 2001 From: JHunniford <97452751+J-Hunniford@users.noreply.github.com> Date: Thu, 7 Dec 2023 15:15:39 +0000 Subject: [PATCH 063/110] another quick edit to clarify primary and db --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 77a8fc65..a3866217 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -28,7 +28,7 @@ bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup. ## Using `recovery` backup and restore -When you run a `recovery` backup plan, the primary server configuration is backed up in full. In the event of a primary server failure, this backup can be used to restore the PE installation to the state it was in when the backup was created. +When you run a `recovery` backup plan, the primary server configuration is backed up in full. In the event of a primary server failure, this backup can be used to to restore your primary server and all services (including database services running on external servers) to the exact state they were in at the time of the backup. You can create a `recovery` backup as follows: ``` From a79db7c34f54b4475b11882490828f0f36ab24c3 Mon Sep 17 00:00:00 2001 From: JHunniford <97452751+J-Hunniford@users.noreply.github.com> Date: Thu, 7 Dec 2023 15:38:21 +0000 Subject: [PATCH 064/110] Another quick edit for clarity --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index a3866217..eb35ad4f 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -7,7 +7,7 @@ If your PE installation is managed by `peadm`, you can back up and restore PE us **Important:** If your PE installation is not managed by `peadm`, you cannot use the `peadm::backup` and `peadm::restore` plans. For information on converting to a `peadm` managed installation, see [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md). When running the backup and restore plans, you can define the `backup_type` and `restore_type` parameters with either of the following values: -* `recovery`: Use this type to create a full backup of your primary server, including data for all services. This allows you to restore your primary server and all services (including database services running on external servers) to the exact state they were in at the time of the backup. +* `recovery`: Use this type to create a full backup of your primary server, including data for all services, including database services running on external servers. This allows you to restore your primary server and all services to the exact state they were in at the time of the backup. * `custom`: Use this type when you want to selectively back up and restore data for specific services. If no type is specified, the default is `recovery`. From e81ab577828eb6e14b2b09d315352edc32718be8 Mon Sep 17 00:00:00 2001 From: JHunniford <97452751+J-Hunniford@users.noreply.github.com> Date: Thu, 7 Dec 2023 15:40:45 +0000 Subject: [PATCH 065/110] revert previous commit --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index eb35ad4f..a3866217 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -7,7 +7,7 @@ If your PE installation is managed by `peadm`, you can back up and restore PE us **Important:** If your PE installation is not managed by `peadm`, you cannot use the `peadm::backup` and `peadm::restore` plans. For information on converting to a `peadm` managed installation, see [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md). When running the backup and restore plans, you can define the `backup_type` and `restore_type` parameters with either of the following values: -* `recovery`: Use this type to create a full backup of your primary server, including data for all services, including database services running on external servers. This allows you to restore your primary server and all services to the exact state they were in at the time of the backup. +* `recovery`: Use this type to create a full backup of your primary server, including data for all services. This allows you to restore your primary server and all services (including database services running on external servers) to the exact state they were in at the time of the backup. * `custom`: Use this type when you want to selectively back up and restore data for specific services. If no type is specified, the default is `recovery`. From 2af77683847027c4b28d98e2af41ef4eda63a318 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko <1586813+timidri@users.noreply.github.com> Date: Thu, 7 Dec 2023 19:15:20 +0100 Subject: [PATCH 066/110] Manually copying Jason's suggestion --- documentation/backup_restore.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index a3866217..da8e40af 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -51,6 +51,8 @@ bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup. To specify the items that are backed up and restored, define the `backup_type` or `restore_type` parameters as `custom`. Otherwise, the default type is `recovery`. +**Note:** To customize the list of items that are backed up and restored, you must define the `backup` and `restore` parameters, specifying the items you want to exclude. + To specify the `custom` items, you can create and reference `params.json` files as shown in the following examples. To specify custom backup options: From 89fa2771ac86c12a6304f86384db22708c769ee9 Mon Sep 17 00:00:00 2001 From: JHunniford <97452751+J-Hunniford@users.noreply.github.com> Date: Mon, 11 Dec 2023 17:37:39 +0000 Subject: [PATCH 067/110] Update backup_restore.md Minor "find and replace" edit to reflect decision to use "PEADM" rather than `peadm`. --- documentation/backup_restore.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index da8e40af..7fc19b89 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -1,10 +1,10 @@ # Backup and restore Puppet Enterprise (PE) -If your PE installation is managed by `peadm`, you can back up and restore PE using this process: +If your PE installation is managed by PEADM, you can back up and restore PE using this process: 1. Use the `peadm::backup` plan to create a backup of your primary server. 2. Use the `peadm::restore` plan to restore PE from a `peadm::backup`. -**Important:** If your PE installation is not managed by `peadm`, you cannot use the `peadm::backup` and `peadm::restore` plans. For information on converting to a `peadm` managed installation, see [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md). +**Important:** If your PE installation is not managed by PEADM, you cannot use the `peadm::backup` and `peadm::restore` plans. For information on converting to a PEADM-managed installation, see [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md). When running the backup and restore plans, you can define the `backup_type` and `restore_type` parameters with either of the following values: * `recovery`: Use this type to create a full backup of your primary server, including data for all services. This allows you to restore your primary server and all services (including database services running on external servers) to the exact state they were in at the time of the backup. @@ -116,6 +116,6 @@ The following table shows the items you can specify and indicates what is includ | `puppetdb` | PuppetDB database (including support for XL where puppetdb is running on an external db server) | ✅ | | `rbac` | RBAC database | | -**Note**: The `peadm` backup and restore plans utilize the `puppet-backup` tool for backing up and restoring `ca`, `code` and `config`. For `config`, the data backed up includes the `activity`, `classifier`, `orchestrator`, and `rbac` databases. +**Note**: The PEADM backup and restore plans utilize the `puppet-backup` tool for backing up and restoring `ca`, `code` and `config`. For `config`, the data backed up includes the `activity`, `classifier`, `orchestrator`, and `rbac` databases. **Note:** The output for the `peadm::backup` plan differs from the output that is returned when you manually run the [`puppet-backup create` command](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure). From ca990e33c91af4a05858b05dccf7cf2513a7e3f6 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 8 Jan 2024 19:08:08 +0100 Subject: [PATCH 068/110] Added todos --- .github/workflows/test-backup-restore.yaml | 10 ++++++++-- spec/acceptance/peadm_spec/plans/test_restore.pp | 3 +++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index f22fa6ab..34af98e2 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -130,8 +130,8 @@ jobs: bundle exec bolt command run "rm -rf /etc/puppetlabs/puppetserver/ca /etc/puppetlabs/puppet/ssl" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml echo "Removing classifier database" bundle exec bolt command run "rm -rf /opt/puppetlabs/server/data/postgresql/classifier" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + #TODO if arch is XL, run pe-uninstaller on the primary database - name: Reinstall PE primary in a non-peadm way continue-on-error: true @@ -142,7 +142,7 @@ jobs: --modulepath spec/fixtures/modules --verbose --stream - - name: Perform peadm restore of cluster + - name: Perform peadm restore of primary server timeout-minutes: 30 continue-on-error: true run: | @@ -161,6 +161,12 @@ jobs: --stream echo ::endgroup:: + #TODO new step to run only if arch is XL + # - clean the primary db certificate on the primary server (puppet ca clean --certname ) + # - run the pe installer on the primary db server + # - run the add_database plan on the primary db server + # - run the peadm restore plan with the option "recover_primary_db" set to true + # add docs to README about actions the customer needs to do to recover the primary db - name: Output PE cluster status run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index c42dc46b..fe9368ea 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -1,3 +1,6 @@ +#TODO parametrize the plan so it can do: +# - a recovery restore of the primary server +# - a recovery restore of the primary db server plan peadm_spec::test_restore() { $t = get_targets('*') wait_until_available($t) From a925ad11ee295eb23b021f2043b446806e7ef3ef Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Wed, 10 Jan 2024 17:59:23 +0100 Subject: [PATCH 069/110] added recovery-db restore type --- plans/restore.pp | 15 +++++++++------ spec/acceptance/peadm_spec/plans/test_restore.pp | 8 ++++++-- 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/plans/restore.pp b/plans/restore.pp index 7fba2a12..d508c110 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -1,7 +1,7 @@ # @summary Restore puppet primary configuration # # @param targets These are a list of the primary puppetservers from one or multiple puppet clusters -# @param backup_type Currently, the `recovery` and `custom` restore types are supported +# @param restore_type Choose from `recovery`, `recovery-db` and `custom` # @param restore A hash of custom backup options, see the peadm::recovery_opts_default() function for the default values # @param input_file The file containing the backup to restore from # @example @@ -12,10 +12,10 @@ Peadm::SingleTargetSpec $targets, # restore type determines the restore options - Enum['recovery', 'custom'] $restore_type = 'recovery', + Enum['recovery', 'recovery-db', 'custom'] $restore_type = 'recovery', # Which data to restore - Peadm::Recovery_opts $restore = {}, + Peadm::Recovery_opts $restore = {}, # Path to the recovery tarball Pattern[/.*\.tar\.gz$/] $input_file, @@ -58,9 +58,10 @@ ) $recovery_opts = $restore_type? { - 'recovery' => peadm::recovery_opts_default(), - 'migration' => peadm::migration_opts_default(), - 'custom' => peadm::migration_opts_all() + $restore, + 'recovery' => peadm::recovery_opts_default(), + 'recovery-db' => { 'puppetdb' => true, }, + 'migration' => peadm::migration_opts_default(), + 'custom' => peadm::migration_opts_all() + $restore, } $primary_target = peadm::get_targets(getvar('cluster.params.primary_host'), 1) @@ -125,6 +126,8 @@ ${shellquote($recovery_directory)}/recovery/pe_backup-*tgz | CMD # lint:endignore + } elsif $restore_type == 'recovery-db' { + out::message('# Restoring primary database for recovery') } else { if getvar('recovery_opts.ca') { out::message('# Restoring ca and ssl certificates') diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index fe9368ea..39466d01 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -1,7 +1,11 @@ #TODO parametrize the plan so it can do: # - a recovery restore of the primary server # - a recovery restore of the primary db server -plan peadm_spec::test_restore() { +plan peadm_spec::test_restore( + # restore type determines the restore options + Enum['recovery', 'recovery-db'] $restore_type = 'recovery', + +) { $t = get_targets('*') wait_until_available($t) @@ -26,7 +30,7 @@ $result = run_command('ls -t /tmp/pe-backup*gz | head -1', $primary_host).first.value $input_file = strip(getvar('result.stdout')) - run_plan('peadm::restore', $primary_host, { 'restore_type' => 'recovery', 'input_file' => $input_file }) + run_plan('peadm::restore', $primary_host, { 'restore_type' => restore_type, 'input_file' => $input_file }) # run infra status on the primary out::message("Running peadm::status on primary host ${primary_host}") From 5dca665c3cf2577c2159bffcbca36d4be236f2b3 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Thu, 11 Jan 2024 16:29:06 +0000 Subject: [PATCH 070/110] Updating workflow for db recovery --- .github/workflows/test-backup-restore.yaml | 54 ++++++++++++++++++- .../peadm_spec/plans/test_restore.pp | 2 +- 2 files changed, 53 insertions(+), 3 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 34af98e2..9165ef18 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -19,6 +19,14 @@ on: description: "PE architecture to test" required: true default: "standard" + type: choice + options: + - standard + - standard-with-dr + - large + - large-with-dr + - extra-large + - extra-large-with-dr version: description: "PE version to install" required: true @@ -133,14 +141,25 @@ jobs: bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml #TODO if arch is XL, run pe-uninstaller on the primary database + if [ ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' ]; then + echo "Uninstalling PE from primary database" + primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt task run peadm_spec::reinstall_pe version=${{ github.event.inputs.version || '2023.5.0' }} -t $primary_db \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules \ + --verbose \ + --stream + fi + - name: Reinstall PE primary in a non-peadm way continue-on-error: true run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt task run peadm_spec::reinstall_pe version=${{ github.event.inputs.version || '2023.5.0' }} -t $primary \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ - --modulepath spec/fixtures/modules - --verbose --stream + --modulepath spec/fixtures/modules \ + --verbose \ + --stream - name: Perform peadm restore of primary server timeout-minutes: 30 @@ -154,6 +173,9 @@ jobs: bundle exec rake spec_prep echo ::endgroup:: + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt command run "echo '[Service]' | sudo tee /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'TimeoutStartSec=1' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'TimeoutStopSec=1' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'Restart=no' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; sudo systemctl daemon-reload" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + echo ::group::restore bundle exec bolt plan run peadm_spec::test_restore \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ @@ -161,12 +183,40 @@ jobs: --stream echo ::endgroup:: + bundle exec bolt command run "systemctl stop pe-puppetdb.service && rm /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf && systemctl daemon-reload" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + #TODO new step to run only if arch is XL # - clean the primary db certificate on the primary server (puppet ca clean --certname ) # - run the pe installer on the primary db server # - run the add_database plan on the primary db server # - run the peadm restore plan with the option "recover_primary_db" set to true # add docs to README about actions the customer needs to do to recover the primary db + + - name: Setup Primary DB in XL + if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' + run: | + primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) + db_hostname=$(bundle exec bolt command run "hostname -f" -t $primary_db --inventoryfile spec/fixtures/litmus_inventory.yaml | grep -oP 'litmus-\w+\.c\.ia-content\.internal') + primary_hostname=$(bundle exec bolt command run "hostname -f" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml | grep -oP 'litmus-\w+\.c\.ia-content\.internal') + bundle exec bolt command run "puppetserver ca clean --certname $db_hostname" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + # bundle exec bolt plan run peadm::add_database -t $primary_db primary_host=$primary \ + # --inventoryfile spec/fixtures/litmus_inventory.yaml \ + # --modulepath spec/fixtures/modules + + bundle exec bolt plan run peadm::subplans::component_install -t $primary_db \ + primary_host=$primary_hostname \ + avail_group_letter='A' \ + role='puppet/puppetdb-database' \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules \ + --stream + + bundle exec bolt plan run peadm_spec::test_restore restore_type="recovery-db" \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules \ + --stream + - name: Output PE cluster status run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) diff --git a/spec/acceptance/peadm_spec/plans/test_restore.pp b/spec/acceptance/peadm_spec/plans/test_restore.pp index 39466d01..d09e84d8 100644 --- a/spec/acceptance/peadm_spec/plans/test_restore.pp +++ b/spec/acceptance/peadm_spec/plans/test_restore.pp @@ -30,7 +30,7 @@ $result = run_command('ls -t /tmp/pe-backup*gz | head -1', $primary_host).first.value $input_file = strip(getvar('result.stdout')) - run_plan('peadm::restore', $primary_host, { 'restore_type' => restore_type, 'input_file' => $input_file }) + run_plan('peadm::restore', $primary_host, { 'restore_type' => $restore_type, 'input_file' => $input_file }) # run infra status on the primary out::message("Running peadm::status on primary host ${primary_host}") From a01eb434fceb1eee095b9ccbe479147f7aa6a3b3 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 29 Jan 2024 17:29:12 +0100 Subject: [PATCH 071/110] Add init_db_server and refactor --- .github/workflows/test-backup-restore.yaml | 17 ++-- plans/restore.pp | 4 + .../peadm_spec/plans/init_db_server.pp | 82 +++++++++++++++++++ 3 files changed, 91 insertions(+), 12 deletions(-) create mode 100644 spec/acceptance/peadm_spec/plans/init_db_server.pp diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 9165ef18..9f78254e 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -144,7 +144,7 @@ jobs: if [ ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' ]; then echo "Uninstalling PE from primary database" primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt task run peadm_spec::reinstall_pe version=${{ github.event.inputs.version || '2023.5.0' }} -t $primary_db \ + bundle exec bolt command run "/opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d -y || true" -t $primary_db \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ --verbose \ @@ -196,18 +196,11 @@ jobs: if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' run: | primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) - primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) db_hostname=$(bundle exec bolt command run "hostname -f" -t $primary_db --inventoryfile spec/fixtures/litmus_inventory.yaml | grep -oP 'litmus-\w+\.c\.ia-content\.internal') - primary_hostname=$(bundle exec bolt command run "hostname -f" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml | grep -oP 'litmus-\w+\.c\.ia-content\.internal') - bundle exec bolt command run "puppetserver ca clean --certname $db_hostname" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - # bundle exec bolt plan run peadm::add_database -t $primary_db primary_host=$primary \ - # --inventoryfile spec/fixtures/litmus_inventory.yaml \ - # --modulepath spec/fixtures/modules - - bundle exec bolt plan run peadm::subplans::component_install -t $primary_db \ - primary_host=$primary_hostname \ - avail_group_letter='A' \ - role='puppet/puppetdb-database' \ + # primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) + # primary_hostname=$(bundle exec bolt command run "hostname -f" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml | grep -oP 'litmus-\w+\.c\.ia-content\.internal') + + bundle exec bolt plan run peadm_spec::init_db_server db_host=$db_hostname \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ --stream diff --git a/plans/restore.pp b/plans/restore.pp index d508c110..70d432f8 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -300,6 +300,10 @@ # Run Puppet to pick up last remaining config tweaks run_task('peadm::puppet_runonce', $primary_target) + if $restore_type == 'recovery-db' { + run_task('peadm::puppet_runonce', $puppetdb_postgresql_targets) + } + apply($primary_target) { file { $recovery_directory : ensure => 'absent', diff --git a/spec/acceptance/peadm_spec/plans/init_db_server.pp b/spec/acceptance/peadm_spec/plans/init_db_server.pp new file mode 100644 index 00000000..f4441341 --- /dev/null +++ b/spec/acceptance/peadm_spec/plans/init_db_server.pp @@ -0,0 +1,82 @@ +plan peadm_spec::init_db_server( + Peadm::SingleTargetSpec $db_host, + String[1] $pe_version = '2023.4.0', + String[1] $pe_platform = 'el-8-x86_64', +) { + $t = get_targets('*') + wait_until_available($t) + + $db_target = get_target($db_host) + parallelize($t + $db_target) |$target| { + $fqdn = run_command('hostname -f', $target) + $target.set_var('certname', $fqdn.first['stdout'].chomp) + } + + $primary_target = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] + $compiler_targets = $t.filter |$n| { $n.vars['role'] == 'compiler' } + + out::message("db_target: ${db_target}") + out::message("db_target certname: ${db_target.peadm::certname()}") + out::message("primary_target: ${primary_target}") + out::message("compiler_targets: ${compiler_targets}") + # fail_plan('testing eop') + run_command("/opt/puppetlabs/bin/puppetserver ca clean --certname ${db_target.peadm::certname()}", $primary_target) + + $pe_conf_data = {} + + $puppetdb_database_temp_config = { + 'puppet_enterprise::profile::database::puppetdb_hosts' => ( + $compiler_targets + $primary_target + ).map |$t| { $t.peadm::certname() }, + } + + $primary_postgresql_pe_conf = peadm::generate_pe_conf({ + 'console_admin_password' => 'not used', + 'puppet_enterprise::puppet_master_host' => $primary_target.peadm::certname(), + 'puppet_enterprise::database_host' => $db_target.peadm::certname(), + } + $puppetdb_database_temp_config + $pe_conf_data) + + # Upload the pe.conf files to the hosts that need them, and ensure correctly + # configured certnames. Right now for these hosts we need to do that by + # staging a puppet.conf file. + + peadm::file_content_upload($primary_postgresql_pe_conf, '/tmp/pe.conf', $db_target) +# lint:ignore:strict_indent + run_task('peadm::mkdir_p_file', $db_target, + path => '/etc/puppetlabs/puppet/puppet.conf', + content => @("HEREDOC"), + [main] + certname = ${db_target.peadm::certname()} + | HEREDOC +# lint:endignore + ) + + run_plan('peadm::util::insert_csr_extension_requests', $db_target, + extension_requests => { + peadm::oid('peadm_role') => 'puppet/puppetdb-database', + peadm::oid('peadm_availability_group') => 'A', + }) + + $uploaddir = '/tmp' + $pe_tarball_name = "puppet-enterprise-${pe_version}-${pe_platform}.tar.gz" + $pe_tarball_source = "https://s3.amazonaws.com/pe-builds/released/${pe_version}/${pe_tarball_name}" + $upload_tarball_path = "${uploaddir}/${pe_tarball_name}" + + run_task('peadm::download', $db_target, + source => $pe_tarball_source, + path => $upload_tarball_path, + ) + + # Run the PE installer on the puppetdb database hosts + run_task('peadm::pe_install', $db_target, + tarball => $upload_tarball_path, + peconf => '/tmp/pe.conf', + puppet_service_ensure => 'stopped', + ) + + run_plan('peadm::subplans::component_install', $db_target, { + primary_host => $primary_target.peadm::certname(), + avail_group_letter => 'A', + role => 'puppet/puppetdb-database', + }) +} From 782fd1d88c632494783a3bb0fe918cb5db938a92 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 29 Jan 2024 19:36:04 +0100 Subject: [PATCH 072/110] fix error handling --- .github/workflows/test-backup-restore.yaml | 18 ++++++------------ .../peadm_spec/plans/init_db_server.pp | 2 +- 2 files changed, 7 insertions(+), 13 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 9f78254e..68312926 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -180,30 +180,24 @@ jobs: bundle exec bolt plan run peadm_spec::test_restore \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ - --stream + --stream \ + || true # ignore errors echo ::endgroup:: bundle exec bolt command run "systemctl stop pe-puppetdb.service && rm /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf && systemctl daemon-reload" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - #TODO new step to run only if arch is XL - # - clean the primary db certificate on the primary server (puppet ca clean --certname ) - # - run the pe installer on the primary db server - # - run the add_database plan on the primary db server - # - run the peadm restore plan with the option "recover_primary_db" set to true - # add docs to README about actions the customer needs to do to recover the primary db + # TODO: add docs to README about actions the customer needs to do to recover the primary db - name: Setup Primary DB in XL if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' run: | primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) - db_hostname=$(bundle exec bolt command run "hostname -f" -t $primary_db --inventoryfile spec/fixtures/litmus_inventory.yaml | grep -oP 'litmus-\w+\.c\.ia-content\.internal') - # primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) - # primary_hostname=$(bundle exec bolt command run "hostname -f" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml | grep -oP 'litmus-\w+\.c\.ia-content\.internal') - bundle exec bolt plan run peadm_spec::init_db_server db_host=$db_hostname \ + bundle exec bolt plan run peadm_spec::init_db_server db_host=$primary_db \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ - --stream + --stream \ + || true # ignore errors bundle exec bolt plan run peadm_spec::test_restore restore_type="recovery-db" \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ diff --git a/spec/acceptance/peadm_spec/plans/init_db_server.pp b/spec/acceptance/peadm_spec/plans/init_db_server.pp index f4441341..a21b2cc7 100644 --- a/spec/acceptance/peadm_spec/plans/init_db_server.pp +++ b/spec/acceptance/peadm_spec/plans/init_db_server.pp @@ -75,7 +75,7 @@ ) run_plan('peadm::subplans::component_install', $db_target, { - primary_host => $primary_target.peadm::certname(), + primary_host => $primary_target, avail_group_letter => 'A', role => 'puppet/puppetdb-database', }) From 7b080dff9650d5754e6a1896a4a063a0e17afe7c Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 13 Feb 2024 20:01:50 +0100 Subject: [PATCH 073/110] document changes needed --- .github/workflows/test-backup-restore.yaml | 25 +++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 68312926..08f26c22 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -176,6 +176,9 @@ jobs: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt command run "echo '[Service]' | sudo tee /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'TimeoutStartSec=1' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'TimeoutStopSec=1' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'Restart=no' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; sudo systemctl daemon-reload" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + #TODO update the restore to: + # - restore the puppetdb in the standard cases + # - not restore the puppetdb if there is a broken external db echo ::group::restore bundle exec bolt plan run peadm_spec::test_restore \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ @@ -193,12 +196,24 @@ jobs: run: | primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt plan run peadm_spec::init_db_server db_host=$primary_db \ - --inventoryfile spec/fixtures/litmus_inventory.yaml \ - --modulepath spec/fixtures/modules \ - --stream \ - || true # ignore errors + # bundle exec bolt plan run peadm_spec::init_db_server db_host=$primary_db \ + # --inventoryfile spec/fixtures/litmus_inventory.yaml \ + # --modulepath spec/fixtures/modules \ + # --stream \ + # || true # ignore errors + + #TODO: update peadm configuration by running the plan peadm::util::update_classification + # specifying the restored peadm config except providing "undef" for both database hosts + # (this will initialise puppetdb on the primary) + # should this ^^^ be part of the restore plan with recovery-db restore type? + + #TODO: run peadm::add_database on the primary db + # after this step, we expect this to be the state: + # - the primary (external) postgresql server is up and running + # - puppet primary points to the external server for its puppetdb + + # restore the puppetdb database bundle exec bolt plan run peadm_spec::test_restore restore_type="recovery-db" \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ From b0de776b416cc831ae61c3cf3f95ef710208637f Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Mon, 19 Feb 2024 14:09:49 +0000 Subject: [PATCH 074/110] Updating db recovery step --- .github/workflows/test-backup-restore.yaml | 3 +-- plans/restore.pp | 13 +++++++++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 08f26c22..81853396 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -194,8 +194,7 @@ jobs: - name: Setup Primary DB in XL if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' run: | - primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) - + # primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) # bundle exec bolt plan run peadm_spec::init_db_server db_host=$primary_db \ # --inventoryfile spec/fixtures/litmus_inventory.yaml \ diff --git a/plans/restore.pp b/plans/restore.pp index 70d432f8..ec6a62e6 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -128,6 +128,19 @@ # lint:endignore } elsif $restore_type == 'recovery-db' { out::message('# Restoring primary database for recovery') + # lint:ignore:strict_indent + run_plan('peadm::util::update_classification', $primary_target, + postgresql_a_host => undef, + postgresql_b_host => undef, + peadm_config => $cluster + ) + # lint:endignore + + run_task('peadm::puppet_runonce', $primary_target) + + run_plan('peadm::add_database', getvar('cluster.params.primary_postgresql_host'), primary_host => $primary_target) + + # should db data restore be done here? } else { if getvar('recovery_opts.ca') { out::message('# Restoring ca and ssl certificates') From f9a8751de42c82a38eae12626daa7779571f22e1 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Wed, 6 Mar 2024 14:57:14 +0000 Subject: [PATCH 075/110] Updating db restore --- plans/backup.pp | 8 +++ plans/restore.pp | 52 +++++++++++++++---- plans/util/sanitize_pg_pe_conf.pp | 10 ++-- .../peadm_spec/tasks/reinstall_pe.json | 5 ++ .../peadm_spec/tasks/reinstall_pe.sh | 4 +- 5 files changed, 65 insertions(+), 14 deletions(-) diff --git a/plans/backup.pp b/plans/backup.pp index 493fab35..94606709 100644 --- a/plans/backup.pp +++ b/plans/backup.pp @@ -84,6 +84,14 @@ mode => '0600', } + # backup the recovery options + file { "${backup_directory}/peadm/recovery_opts.json": + content => stdlib::to_json_pretty($recovery_opts), + owner => 'root', + group => 'root', + mode => '0600', + } + # Create a subdir for each backup type selected $recovery_opts.filter |$_,$val| { $val == true }.each |$dir,$_| { file { "${backup_directory}/${dir}": diff --git a/plans/restore.pp b/plans/restore.pp index ec6a62e6..6fbf636a 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -1,11 +1,11 @@ # @summary Restore puppet primary configuration # -# @param targets These are a list of the primary puppetservers from one or multiple puppet clusters +# @param targets This should be the primary puppetserver for the puppet cluster # @param restore_type Choose from `recovery`, `recovery-db` and `custom` # @param restore A hash of custom backup options, see the peadm::recovery_opts_default() function for the default values # @param input_file The file containing the backup to restore from # @example -# bolt plan run peadm::restore -t primary1.example.com,primary2.example.com input_file=/tmp/peadm-backup.tar.gz +# bolt plan run peadm::restore -t primary1.example.com input_file=/tmp/peadm-backup.tar.gz # plan peadm::restore ( # This plan should be run on the primary server @@ -33,17 +33,21 @@ # try to load the cluster configuration by running peadm::get_peadm_config, but allow for errors to happen $_cluster = run_task('peadm::get_peadm_config', $targets, { '_catch_errors' => true }).first.value - out::message("cluster: ${_cluster}") + + $result = download_file("${recovery_directory}/peadm/peadm_config.json", 'peadm_config.json', $targets).first.value + $cluster_backup = loadjson(getvar('result.path')) + if $_cluster == undef or getvar('_cluster.params') == undef { # failed to get cluster config, load from backup out::message('Failed to get cluster configuration, loading from backup...') - $result = download_file("${recovery_directory}/peadm/peadm_config.json", 'peadm_config.json', $targets).first.value - $cluster = loadjson(getvar('result.path')) + $cluster = $cluster_backup out::message('Cluster configuration loaded from backup') } else { $cluster = $_cluster } + out::message("cluster: ${cluster}") + $error = getvar('cluster.error') if $error { fail_plan($error) @@ -130,18 +134,46 @@ out::message('# Restoring primary database for recovery') # lint:ignore:strict_indent run_plan('peadm::util::update_classification', $primary_target, - postgresql_a_host => undef, - postgresql_b_host => undef, + postgresql_a_host => getvar('cluster.params.primary_host'), peadm_config => $cluster ) # lint:endignore + run_plan('peadm::util::sanitize_pe_conf', $primary_target) + # lint:ignore:strict_indent + run_command(@("CMD"/L), $primary_target) + rm -f /etc/puppetlabs/puppet/user_data.conf + | CMD + # lint:endignore + + # lint:ignore:strict_indent + run_command(@("CMD"/L), $primary_target) + systemctl stop pe-puppetdb + | CMD + # lint:endignore + + # lint:ignore:strict_indent + run_command(@("CMD"/L), $primary_target) + puppet infra configure + | CMD + # lint:endignore + + run_task('peadm::puppet_runonce', $primary_target) run_task('peadm::puppet_runonce', $primary_target) - run_plan('peadm::add_database', getvar('cluster.params.primary_postgresql_host'), primary_host => $primary_target) + run_plan('peadm::add_database', getvar('cluster_backup.params.primary_postgresql_host'), primary_host => $primary_target) - # should db data restore be done here? - } else { + # restore from snapshot + # - run classification update + # - run sanitize pe.conf + # - remove user_data.conf + # - stop puppetdb + # - run puppet infrastructure configure + # - do 2 puppet agent runs + # - add db + + # fail_plan('Testing db restore') +} else { if getvar('recovery_opts.ca') { out::message('# Restoring ca and ssl certificates') # lint:ignore:strict_indent diff --git a/plans/util/sanitize_pg_pe_conf.pp b/plans/util/sanitize_pg_pe_conf.pp index 5888bd09..c3033bd0 100644 --- a/plans/util/sanitize_pg_pe_conf.pp +++ b/plans/util/sanitize_pg_pe_conf.pp @@ -11,11 +11,15 @@ run_task('peadm::read_file', $targets, path => $path, ).map |$result| { - $sanitized = $result['content'].loadjson() + { + out::message("Sanitizing pe.conf on ${result.value}") + $sanitized = $result.value['content'].parsejson() + { 'puppet_enterprise::puppet_master_host' => $primary_target.peadm::certname(), - 'puppet_enterprise::database_host' => $result.target.peadm::certname(), + 'puppet_enterprise::puppetdb_database_host' => $result.target.peadm::certname(), } + + out::message("Sanitized ${sanitized}") + # Return the result of file_content_upload. There is only one target - peadm::file_content_upload($sanitized, $path, $result.target)[0] + peadm::file_content_upload(stdlib::to_json_pretty($sanitized), $path, $result.target)[0] } } diff --git a/spec/acceptance/peadm_spec/tasks/reinstall_pe.json b/spec/acceptance/peadm_spec/tasks/reinstall_pe.json index fb2ccd33..2d83fa72 100644 --- a/spec/acceptance/peadm_spec/tasks/reinstall_pe.json +++ b/spec/acceptance/peadm_spec/tasks/reinstall_pe.json @@ -9,6 +9,11 @@ "type": "String[1]", "description": "The PE installation platform", "default": "el-8-x86_64" + }, + "uninstall": { + "type": "Boolean", + "description": "Whether we want to uninstall PE before installing", + "default": false } } } diff --git a/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh b/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh index 5175a63e..a7d8a4da 100644 --- a/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh +++ b/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh @@ -2,7 +2,9 @@ # This task reinstalls PE and needs to run as root. # Uninstall PE if installed -/opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d -y || true +if [[ "$PT_uninstall" == true ]]; then + /opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d -y || true +fi # Download PE INSTALLER="puppet-enterprise-${PT_version}-${PT_arch}" From 621e3d55ecf7f572b8500920d1d612586bdc2228 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 8 Mar 2024 15:35:58 +0100 Subject: [PATCH 076/110] revert back to running puppet installer on the db server use init_db_server plan inside restore --- plans/restore.pp | 46 +++---------------- .../peadm_spec/plans/init_db_server.pp | 9 +++- 2 files changed, 13 insertions(+), 42 deletions(-) diff --git a/plans/restore.pp b/plans/restore.pp index 6fbf636a..e830b97f 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -132,48 +132,14 @@ # lint:endignore } elsif $restore_type == 'recovery-db' { out::message('# Restoring primary database for recovery') - # lint:ignore:strict_indent - run_plan('peadm::util::update_classification', $primary_target, - postgresql_a_host => getvar('cluster.params.primary_host'), - peadm_config => $cluster - ) - # lint:endignore - - run_plan('peadm::util::sanitize_pe_conf', $primary_target) - # lint:ignore:strict_indent - run_command(@("CMD"/L), $primary_target) - rm -f /etc/puppetlabs/puppet/user_data.conf - | CMD - # lint:endignore - - # lint:ignore:strict_indent - run_command(@("CMD"/L), $primary_target) - systemctl stop pe-puppetdb - | CMD - # lint:endignore - - # lint:ignore:strict_indent - run_command(@("CMD"/L), $primary_target) - puppet infra configure - | CMD - # lint:endignore - - run_task('peadm::puppet_runonce', $primary_target) - run_task('peadm::puppet_runonce', $primary_target) - - run_plan('peadm::add_database', getvar('cluster_backup.params.primary_postgresql_host'), primary_host => $primary_target) - - # restore from snapshot - # - run classification update - # - run sanitize pe.conf - # - remove user_data.conf - # - stop puppetdb - # - run puppet infrastructure configure - # - do 2 puppet agent runs - # - add db + run_plan('peadm_spec::init_db_server', { + 'db_host' => getvar('cluster.params.primary_postgresql_host'), + _catch_errors => true + } + ) # fail_plan('Testing db restore') -} else { + } else { if getvar('recovery_opts.ca') { out::message('# Restoring ca and ssl certificates') # lint:ignore:strict_indent diff --git a/spec/acceptance/peadm_spec/plans/init_db_server.pp b/spec/acceptance/peadm_spec/plans/init_db_server.pp index a21b2cc7..0f05bace 100644 --- a/spec/acceptance/peadm_spec/plans/init_db_server.pp +++ b/spec/acceptance/peadm_spec/plans/init_db_server.pp @@ -1,6 +1,6 @@ plan peadm_spec::init_db_server( - Peadm::SingleTargetSpec $db_host, - String[1] $pe_version = '2023.4.0', + String[1] $db_host, + String[1] $pe_version = '2023.5.0', String[1] $pe_platform = 'el-8-x86_64', ) { $t = get_targets('*') @@ -67,6 +67,9 @@ path => $upload_tarball_path, ) + run_task('service', $compiler_targets, { action => 'stop', name => 'pe-puppetdb', _catch_errors => true }) + run_task('service', $primary_target, { action => 'restart', name => 'pe-puppetdb', _catch_errors => true }) + # Run the PE installer on the puppetdb database hosts run_task('peadm::pe_install', $db_target, tarball => $upload_tarball_path, @@ -79,4 +82,6 @@ avail_group_letter => 'A', role => 'puppet/puppetdb-database', }) + run_task('service', $compiler_targets, { action => 'start', name => 'pe-puppetdb', _catch_errors => true }) + run_task('service', $compiler_targets, { action => 'restart', name => 'pe-puppetserver', _catch_errors => true }) } From 32f6c5fb60e8007eb8290d8e59c00b564dc76fc8 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 12 Mar 2024 15:00:21 +0100 Subject: [PATCH 077/110] * add node_manager to bolt-project * parametrize init_db_server * make installing optional --- Puppetfile | 11 ++++ bolt-project.yaml | 9 +++ plans/restore.pp | 4 +- .../peadm_spec/plans/init_db_server.pp | 61 ++++++++++--------- 4 files changed, 56 insertions(+), 29 deletions(-) create mode 100644 Puppetfile create mode 100644 bolt-project.yaml diff --git a/Puppetfile b/Puppetfile new file mode 100644 index 00000000..d304ab39 --- /dev/null +++ b/Puppetfile @@ -0,0 +1,11 @@ +# This Puppetfile is managed by Bolt. Do not edit. +# For more information, see https://pup.pt/bolt-modules + +# The following directive installs modules to the managed moduledir. +moduledir '.modules' + +mod 'node_manager', + git: 'https://github.com/WhatsARanjit/puppet-node_manager', + ref: '86cb48f27a4a0ea212b33f86775d84a374a189b7' +mod 'puppet/format', '1.1.1' +mod 'puppetlabs/stdlib', '9.5.0' diff --git a/bolt-project.yaml b/bolt-project.yaml new file mode 100644 index 00000000..306fd6ef --- /dev/null +++ b/bolt-project.yaml @@ -0,0 +1,9 @@ +--- +name: peadm +stream: true +modules: + - puppetlabs/stdlib + - puppet/format + - name: node_manager + git: https://github.com/WhatsARanjit/puppet-node_manager + ref: master diff --git a/plans/restore.pp b/plans/restore.pp index e830b97f..7c3c6aef 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -133,7 +133,9 @@ } elsif $restore_type == 'recovery-db' { out::message('# Restoring primary database for recovery') run_plan('peadm_spec::init_db_server', { - 'db_host' => getvar('cluster.params.primary_postgresql_host'), + db_host => getvar('cluster.params.primary_postgresql_host'), + install_pe => true, + pe_version => getvar('cluster.params.pe_version'), _catch_errors => true } ) diff --git a/spec/acceptance/peadm_spec/plans/init_db_server.pp b/spec/acceptance/peadm_spec/plans/init_db_server.pp index 0f05bace..3f54488b 100644 --- a/spec/acceptance/peadm_spec/plans/init_db_server.pp +++ b/spec/acceptance/peadm_spec/plans/init_db_server.pp @@ -1,5 +1,6 @@ plan peadm_spec::init_db_server( String[1] $db_host, + Boolean $install_pe = false, String[1] $pe_version = '2023.5.0', String[1] $pe_platform = 'el-8-x86_64', ) { @@ -22,25 +23,6 @@ # fail_plan('testing eop') run_command("/opt/puppetlabs/bin/puppetserver ca clean --certname ${db_target.peadm::certname()}", $primary_target) - $pe_conf_data = {} - - $puppetdb_database_temp_config = { - 'puppet_enterprise::profile::database::puppetdb_hosts' => ( - $compiler_targets + $primary_target - ).map |$t| { $t.peadm::certname() }, - } - - $primary_postgresql_pe_conf = peadm::generate_pe_conf({ - 'console_admin_password' => 'not used', - 'puppet_enterprise::puppet_master_host' => $primary_target.peadm::certname(), - 'puppet_enterprise::database_host' => $db_target.peadm::certname(), - } + $puppetdb_database_temp_config + $pe_conf_data) - - # Upload the pe.conf files to the hosts that need them, and ensure correctly - # configured certnames. Right now for these hosts we need to do that by - # staging a puppet.conf file. - - peadm::file_content_upload($primary_postgresql_pe_conf, '/tmp/pe.conf', $db_target) # lint:ignore:strict_indent run_task('peadm::mkdir_p_file', $db_target, path => '/etc/puppetlabs/puppet/puppet.conf', @@ -67,21 +49,44 @@ path => $upload_tarball_path, ) - run_task('service', $compiler_targets, { action => 'stop', name => 'pe-puppetdb', _catch_errors => true }) - run_task('service', $primary_target, { action => 'restart', name => 'pe-puppetdb', _catch_errors => true }) + run_command('systemctl stop pe-puppetdb', $compiler_targets, { _catch_errors => true }) + # run_task('service', $primary_target, { action => 'restart', name => 'pe-puppetdb', _catch_errors => true }) - # Run the PE installer on the puppetdb database hosts - run_task('peadm::pe_install', $db_target, - tarball => $upload_tarball_path, - peconf => '/tmp/pe.conf', - puppet_service_ensure => 'stopped', - ) + if $install_pe { + $pe_conf_data = {} + + $puppetdb_database_temp_config = { + 'puppet_enterprise::profile::database::puppetdb_hosts' => ( + $compiler_targets + $primary_target + ).map |$t| { $t.peadm::certname() }, + } + + $primary_postgresql_pe_conf = peadm::generate_pe_conf({ + 'console_admin_password' => 'not used', + 'puppet_enterprise::puppet_master_host' => $primary_target.peadm::certname(), + 'puppet_enterprise::database_host' => $db_target.peadm::certname(), + } + $puppetdb_database_temp_config + $pe_conf_data) + + # Upload the pe.conf files to the hosts that need them, and ensure correctly + # configured certnames. Right now for these hosts we need to do that by + # staging a puppet.conf file. + + peadm::file_content_upload($primary_postgresql_pe_conf, '/tmp/pe.conf', $db_target) + + # Run the PE installer on the puppetdb database hosts + run_task('peadm::pe_install', $db_target, + tarball => $upload_tarball_path, + peconf => '/tmp/pe.conf', + puppet_service_ensure => 'stopped', + ) + } run_plan('peadm::subplans::component_install', $db_target, { primary_host => $primary_target, avail_group_letter => 'A', role => 'puppet/puppetdb-database', }) - run_task('service', $compiler_targets, { action => 'start', name => 'pe-puppetdb', _catch_errors => true }) + run_task('peadm::puppet_runonce', $compiler_targets) + run_command('systemctl start pe-puppetdb', $compiler_targets, { _catch_errors => true }) run_task('service', $compiler_targets, { action => 'restart', name => 'pe-puppetserver', _catch_errors => true }) } From 4aedd184fcb38dd2e8676b9b194d4136afae0b57 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 19 Mar 2024 15:15:56 +0100 Subject: [PATCH 078/110] split recovering primary and database --- .github/workflows/test-backup-restore.yaml | 67 +++++++++++----------- 1 file changed, 33 insertions(+), 34 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 81853396..35da5962 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -37,9 +37,8 @@ on: default: "false" jobs: - backup: - name: "Backup cluster: PE ${{ github.event.inputs.version || '2023.5.0' }} ${{ github.event.inputs.architecture || 'extra-large' }} on ${{ github.event.inputs.image || 'almalinux-cloud/almalinux-8' }}" - + backup-restore-test: + name: "Backup, break and restore cluster: PE ${{ github.event.inputs.version || '2023.5.0' }} ${{ github.event.inputs.architecture || 'extra-large' }} on ${{ github.event.inputs.image || 'almalinux-cloud/almalinux-8' }}" runs-on: ubuntu-20.04 env: BOLT_GEM: true @@ -131,7 +130,7 @@ jobs: with: version: v4.30.5 - - name: Break PE cluster + - name: Break the primary host run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) echo "Removing ssl directories" @@ -141,27 +140,18 @@ jobs: bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml #TODO if arch is XL, run pe-uninstaller on the primary database - if [ ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' ]; then - echo "Uninstalling PE from primary database" - primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt command run "/opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d -y || true" -t $primary_db \ - --inventoryfile spec/fixtures/litmus_inventory.yaml \ - --modulepath spec/fixtures/modules \ - --verbose \ - --stream - fi - - - name: Reinstall PE primary in a non-peadm way + - name: Reinstall PE on the primary host continue-on-error: true + # TODO: move reinstall_pe plan to the peadm::utils folder run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt task run peadm_spec::reinstall_pe version=${{ github.event.inputs.version || '2023.5.0' }} -t $primary \ + bundle exec bolt task run peadm_spec::reinstall_pe uninstall=true version=${{ github.event.inputs.version || '2023.5.0' }} -t $primary \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ --verbose \ --stream - - name: Perform peadm restore of primary server + - name: Perform peadm recovery restore of primary server timeout-minutes: 30 continue-on-error: true run: | @@ -189,28 +179,34 @@ jobs: bundle exec bolt command run "systemctl stop pe-puppetdb.service && rm /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf && systemctl daemon-reload" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + - name: Output PE cluster status + run: | + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + # TODO: add docs to README about actions the customer needs to do to recover the primary db - - name: Setup Primary DB in XL + - name: Break external DB if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' run: | - # primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) - - # bundle exec bolt plan run peadm_spec::init_db_server db_host=$primary_db \ - # --inventoryfile spec/fixtures/litmus_inventory.yaml \ - # --modulepath spec/fixtures/modules \ - # --stream \ - # || true # ignore errors + echo "Uninstalling PE from primary database" + primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt command run "/opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d -y || true" -t $primary_db \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules \ + --verbose \ + --stream - #TODO: update peadm configuration by running the plan peadm::util::update_classification - # specifying the restored peadm config except providing "undef" for both database hosts - # (this will initialise puppetdb on the primary) - # should this ^^^ be part of the restore plan with recovery-db restore type? - - #TODO: run peadm::add_database on the primary db - # after this step, we expect this to be the state: - # - the primary (external) postgresql server is up and running - # - puppet primary points to the external server for its puppetdb + - name: Setup Primary DB in XL + if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' + # TODO: move peadm_spec::init_db_server plan to the peadm::utils folder + run: | + primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt plan run peadm_spec::init_db_server db_host=$primary_db \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules \ + --stream \ + || true # ignore errors # restore the puppetdb database bundle exec bolt plan run peadm_spec::test_restore restore_type="recovery-db" \ @@ -223,6 +219,9 @@ jobs: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + - name: Smoke test + run: "echo 'Smoke test'" + - name: "Wait as long as the file ${HOME}/pause file is present" continue-on-error: true if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} From 718e80060ecd9abdd1ef5907a8c3adc37ca35894 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 19 Mar 2024 15:41:18 +0100 Subject: [PATCH 079/110] remove init_db_server call from the restore plan add smoke test placeholders to workflow --- .github/workflows/test-backup-restore.yaml | 6 ++++++ plans/restore.pp | 11 +---------- 2 files changed, 7 insertions(+), 10 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 35da5962..fda80717 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -184,6 +184,10 @@ jobs: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + # perform a smoke test. Test of "green" puppet infra status is already done insite peadm_spec::test_restore + - name: Smoke test + run: "echo 'Smoke test'" + # TODO: add docs to README about actions the customer needs to do to recover the primary db - name: Break external DB @@ -203,6 +207,7 @@ jobs: run: | primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt plan run peadm_spec::init_db_server db_host=$primary_db \ + install_pe=true \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ --stream \ @@ -219,6 +224,7 @@ jobs: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + # perform a smoke test. Test of "green" puppet infra status is already done insite peadm_spec::test_restore - name: Smoke test run: "echo 'Smoke test'" diff --git a/plans/restore.pp b/plans/restore.pp index 7c3c6aef..6ab6014c 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -132,15 +132,6 @@ # lint:endignore } elsif $restore_type == 'recovery-db' { out::message('# Restoring primary database for recovery') - run_plan('peadm_spec::init_db_server', { - db_host => getvar('cluster.params.primary_postgresql_host'), - install_pe => true, - pe_version => getvar('cluster.params.pe_version'), - _catch_errors => true - } - ) - - # fail_plan('Testing db restore') } else { if getvar('recovery_opts.ca') { out::message('# Restoring ca and ssl certificates') @@ -310,7 +301,7 @@ # lint:endignore } - # Run Puppet to pick up last remaining config tweaks +# Run Puppet to pick up last remaining config tweaks run_task('peadm::puppet_runonce', $primary_target) if $restore_type == 'recovery-db' { From 18a3590a546f79caf9a4ca415a950c8769211b04 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Sun, 24 Mar 2024 13:53:24 +0000 Subject: [PATCH 080/110] Adding puppetrun smoke tests --- .github/workflows/test-backup-restore.yaml | 26 ++++++++++++++++--- .../peadm_spec/plans/puppet_run_test.pp | 22 ++++++++++++++++ 2 files changed, 44 insertions(+), 4 deletions(-) create mode 100644 spec/acceptance/peadm_spec/plans/puppet_run_test.pp diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index fda80717..e73da838 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -140,6 +140,11 @@ jobs: bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml #TODO if arch is XL, run pe-uninstaller on the primary database + - name: Output PE cluster status + run: | + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + - name: Reinstall PE on the primary host continue-on-error: true # TODO: move reinstall_pe plan to the peadm::utils folder @@ -164,7 +169,7 @@ jobs: echo ::endgroup:: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt command run "echo '[Service]' | sudo tee /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'TimeoutStartSec=1' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'TimeoutStopSec=1' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'Restart=no' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; sudo systemctl daemon-reload" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + # bundle exec bolt command run "echo '[Service]' | sudo tee /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'TimeoutStartSec=1' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'TimeoutStopSec=1' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'Restart=no' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; sudo systemctl daemon-reload" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml #TODO update the restore to: # - restore the puppetdb in the standard cases @@ -177,7 +182,7 @@ jobs: || true # ignore errors echo ::endgroup:: - bundle exec bolt command run "systemctl stop pe-puppetdb.service && rm /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf && systemctl daemon-reload" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + # bundle exec bolt command run "systemctl stop pe-puppetdb.service && rm /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf && systemctl daemon-reload && systemctl start pe-puppetdb" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - name: Output PE cluster status run: | @@ -186,7 +191,11 @@ jobs: # perform a smoke test. Test of "green" puppet infra status is already done insite peadm_spec::test_restore - name: Smoke test - run: "echo 'Smoke test'" + run: | + bundle exec bolt plan run peadm_spec::puppet_run_test \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules \ + --stream # TODO: add docs to README about actions the customer needs to do to recover the primary db @@ -201,6 +210,11 @@ jobs: --verbose \ --stream + - name: Output PE cluster status + run: | + primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) + bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml + - name: Setup Primary DB in XL if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' # TODO: move peadm_spec::init_db_server plan to the peadm::utils folder @@ -226,7 +240,11 @@ jobs: # perform a smoke test. Test of "green" puppet infra status is already done insite peadm_spec::test_restore - name: Smoke test - run: "echo 'Smoke test'" + run: | + bundle exec bolt plan run peadm_spec::puppet_run_test \ + --inventoryfile spec/fixtures/litmus_inventory.yaml \ + --modulepath spec/fixtures/modules \ + --stream - name: "Wait as long as the file ${HOME}/pause file is present" continue-on-error: true diff --git a/spec/acceptance/peadm_spec/plans/puppet_run_test.pp b/spec/acceptance/peadm_spec/plans/puppet_run_test.pp new file mode 100644 index 00000000..d6413358 --- /dev/null +++ b/spec/acceptance/peadm_spec/plans/puppet_run_test.pp @@ -0,0 +1,22 @@ +plan peadm_spec::puppet_run_test() { + $t = get_targets('*') + wait_until_available($t) + + parallelize($t) |$target| { + $fqdn = run_command('hostname -f', $target) + $target.set_var('certname', $fqdn.first['stdout'].chomp) + } + + # run puppet on the primary + $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] + out::message("Running puppet on primary host ${primary_host}") + + $status = run_task('peadm::puppet_runonce', $primary_host).first.status + + # Checking for success based on the exit code + if $status == 'success' { + out::message('Puppet run succeeded on the primary host.') + } else { + fail_plan('Puppet run failed on the primary host.') + } +} From 7041b8e63a61c141a06d49e0cb7d6611b10d470f Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Mon, 25 Mar 2024 17:12:34 +0000 Subject: [PATCH 081/110] Moving resinstall and init_db --- .github/workflows/test-backup-restore.yaml | 12 ++---------- .../plans => plans/util}/init_db_server.pp | 2 +- .../peadm_spec/tasks => tasks}/reinstall_pe.json | 2 +- .../peadm_spec/tasks => tasks}/reinstall_pe.sh | 0 4 files changed, 4 insertions(+), 12 deletions(-) rename {spec/acceptance/peadm_spec/plans => plans/util}/init_db_server.pp (99%) rename {spec/acceptance/peadm_spec/tasks => tasks}/reinstall_pe.json (86%) rename {spec/acceptance/peadm_spec/tasks => tasks}/reinstall_pe.sh (100%) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index e73da838..a968befd 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -137,7 +137,6 @@ jobs: bundle exec bolt command run "rm -rf /etc/puppetlabs/puppetserver/ca /etc/puppetlabs/puppet/ssl" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml echo "Removing classifier database" bundle exec bolt command run "rm -rf /opt/puppetlabs/server/data/postgresql/classifier" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml #TODO if arch is XL, run pe-uninstaller on the primary database - name: Output PE cluster status @@ -147,10 +146,9 @@ jobs: - name: Reinstall PE on the primary host continue-on-error: true - # TODO: move reinstall_pe plan to the peadm::utils folder run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt task run peadm_spec::reinstall_pe uninstall=true version=${{ github.event.inputs.version || '2023.5.0' }} -t $primary \ + bundle exec bolt task run peadm::reinstall_pe uninstall=true version=${{ github.event.inputs.version || '2023.5.0' }} -t $primary \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ --verbose \ @@ -169,7 +167,6 @@ jobs: echo ::endgroup:: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) - # bundle exec bolt command run "echo '[Service]' | sudo tee /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'TimeoutStartSec=1' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'TimeoutStopSec=1' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; echo 'Restart=no' | sudo tee -a /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf; sudo systemctl daemon-reload" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml #TODO update the restore to: # - restore the puppetdb in the standard cases @@ -182,14 +179,11 @@ jobs: || true # ignore errors echo ::endgroup:: - # bundle exec bolt command run "systemctl stop pe-puppetdb.service && rm /etc/systemd/system/pe-puppetdb.service.d/10-shortcircuit.conf && systemctl daemon-reload && systemctl start pe-puppetdb" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - - name: Output PE cluster status run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - # perform a smoke test. Test of "green" puppet infra status is already done insite peadm_spec::test_restore - name: Smoke test run: | bundle exec bolt plan run peadm_spec::puppet_run_test \ @@ -217,10 +211,9 @@ jobs: - name: Setup Primary DB in XL if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' - # TODO: move peadm_spec::init_db_server plan to the peadm::utils folder run: | primary_db=$(yq '.groups[].targets[] | select(.vars.role == "primary-pdb-postgresql") | .uri' spec/fixtures/litmus_inventory.yaml) - bundle exec bolt plan run peadm_spec::init_db_server db_host=$primary_db \ + bundle exec bolt plan run peadm::util::init_db_server db_host=$primary_db \ install_pe=true \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ --modulepath spec/fixtures/modules \ @@ -238,7 +231,6 @@ jobs: primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - # perform a smoke test. Test of "green" puppet infra status is already done insite peadm_spec::test_restore - name: Smoke test run: | bundle exec bolt plan run peadm_spec::puppet_run_test \ diff --git a/spec/acceptance/peadm_spec/plans/init_db_server.pp b/plans/util/init_db_server.pp similarity index 99% rename from spec/acceptance/peadm_spec/plans/init_db_server.pp rename to plans/util/init_db_server.pp index 3f54488b..8752803e 100644 --- a/spec/acceptance/peadm_spec/plans/init_db_server.pp +++ b/plans/util/init_db_server.pp @@ -1,4 +1,4 @@ -plan peadm_spec::init_db_server( +plan peadm::util::init_db_server( String[1] $db_host, Boolean $install_pe = false, String[1] $pe_version = '2023.5.0', diff --git a/spec/acceptance/peadm_spec/tasks/reinstall_pe.json b/tasks/reinstall_pe.json similarity index 86% rename from spec/acceptance/peadm_spec/tasks/reinstall_pe.json rename to tasks/reinstall_pe.json index 2d83fa72..2fc81e23 100644 --- a/spec/acceptance/peadm_spec/tasks/reinstall_pe.json +++ b/tasks/reinstall_pe.json @@ -1,5 +1,5 @@ { - "description": "Reinstall PE", + "description": "Reinstall PE, only to be used to restore PE", "parameters": { "version": { "type": "String[1]", diff --git a/spec/acceptance/peadm_spec/tasks/reinstall_pe.sh b/tasks/reinstall_pe.sh similarity index 100% rename from spec/acceptance/peadm_spec/tasks/reinstall_pe.sh rename to tasks/reinstall_pe.sh From 47efb1893f54836f67dee3d3639acc4ac73a59b5 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Thu, 4 Apr 2024 11:26:46 +0100 Subject: [PATCH 082/110] Changing smoke test to run on all hosts --- .github/workflows/test-backup-restore.yaml | 3 +++ .../peadm_spec/plans/puppet_run_test.pp | 20 +++++++++---------- 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index a968befd..1390663b 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -205,6 +205,7 @@ jobs: --stream - name: Output PE cluster status + if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml @@ -227,11 +228,13 @@ jobs: --stream - name: Output PE cluster status + if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' run: | primary=$(yq '.groups[].targets[] | select(.vars.role == "primary") | .uri' spec/fixtures/litmus_inventory.yaml) bundle exec bolt command run "puppet infrastructure status" -t $primary --inventoryfile spec/fixtures/litmus_inventory.yaml - name: Smoke test + if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' run: | bundle exec bolt plan run peadm_spec::puppet_run_test \ --inventoryfile spec/fixtures/litmus_inventory.yaml \ diff --git a/spec/acceptance/peadm_spec/plans/puppet_run_test.pp b/spec/acceptance/peadm_spec/plans/puppet_run_test.pp index d6413358..ceca6329 100644 --- a/spec/acceptance/peadm_spec/plans/puppet_run_test.pp +++ b/spec/acceptance/peadm_spec/plans/puppet_run_test.pp @@ -4,19 +4,17 @@ parallelize($t) |$target| { $fqdn = run_command('hostname -f', $target) - $target.set_var('certname', $fqdn.first['stdout'].chomp) - } + $cert = $target.set_var('certname', $fqdn.first['stdout'].chomp) - # run puppet on the primary - $primary_host = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] - out::message("Running puppet on primary host ${primary_host}") + out::message("Running puppet on host ${cert}.") - $status = run_task('peadm::puppet_runonce', $primary_host).first.status + $status = run_task('peadm::puppet_runonce', $target).first.status - # Checking for success based on the exit code - if $status == 'success' { - out::message('Puppet run succeeded on the primary host.') - } else { - fail_plan('Puppet run failed on the primary host.') + # Checking for success based on the exit code + if $status == 'success' { + out::message("Puppet run succeeded on ${cert}.") + } else { + fail_plan("Puppet run failed on ${cert}.") + } } } From 35232d343427c0f031fafa7db1cf0da1a6a6463e Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Thu, 4 Apr 2024 15:28:58 +0200 Subject: [PATCH 083/110] rewrite spec test --- plans/backup.pp | 8 ++-- spec/plans/backup_spec.rb | 90 ++++++++++++++++++++++++++++++++++----- 2 files changed, 83 insertions(+), 15 deletions(-) diff --git a/plans/backup.pp b/plans/backup.pp index 94606709..df6b52c4 100644 --- a/plans/backup.pp +++ b/plans/backup.pp @@ -38,7 +38,7 @@ $recovery_opts = $backup_type? { 'recovery' => peadm::recovery_opts_default(), 'migration' => peadm::migration_opts_default(), - 'custom' => peadm::migration_opts_all() + $backup, + 'custom' => peadm::recovery_opts_all() + $backup, } $timestamp = Timestamp.new().strftime('%Y-%m-%dT%H%M%SZ') @@ -124,8 +124,7 @@ out::message('# Backing up ca, certs, code and config for recovery') # lint:ignore:strict_indent run_command(@("CMD"), $targets) - /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/recovery \ - --scope=certs,code,config + /opt/puppetlabs/bin/puppet-backup create --dir=${shellquote($backup_directory)}/recovery --scope=certs,code,config | CMD # lint:endignore } else { @@ -169,11 +168,12 @@ if getvar('recovery_opts.orchestrator') { out::message('# Backing up orchestrator secret keys') run_command(@("CMD"), $targets) - cp -rp /etc/puppetlabs/orchestration-services/conf.d/secrets ${shellquote($backup_directory)}/orchestrator/ + cp -rp /etc/puppetlabs/orchestration-services/conf.d/secrets ${shellquote($backup_directory)}/orchestrator/ | CMD } # lint:endignore $backup_databases.each |$name,$database_target| { + out::message("# Backing up database pe-${shellquote($name)}") run_command(@("CMD"/L), $targets) /opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 \ -f ${shellquote($backup_directory)}/${shellquote($name)}/pe-${shellquote($name)}.dump.d \ diff --git a/spec/plans/backup_spec.rb b/spec/plans/backup_spec.rb index f16caecc..113e269b 100644 --- a/spec/plans/backup_spec.rb +++ b/spec/plans/backup_spec.rb @@ -2,21 +2,89 @@ describe 'peadm::backup' do include BoltSpec::Plans - let(:params) { { 'targets' => 'primary' } } + let(:default_params) { { 'targets' => 'primary', 'backup_type' => 'recovery' } } + let(:classifier_only) { + { + 'targets' => 'primary', + 'backup_type' => 'custom', + 'backup' => { + 'activity' => false, + 'ca' => false, + 'classifier' => true, + 'code' => false, + 'config' => false, + 'orchestrator' => false, + 'puppetdb' => false, + 'rbac' => false, + } + } + } + let(:all_backup_options) { + { + 'targets' => 'primary', + 'backup_type' => 'custom', + 'backup' => {} # set all to true + } + } + + before(:each) do + # define a zero timestamp + mocktime = Puppet::Pops::Time::Timestamp.new(0) + # mock the timestamp to always return the zero timestamp + # so the directory name is always the same - /tmp/pe-backup-1970-01-01T000000Z + allow(Puppet::Pops::Time::Timestamp).to receive(:now).and_return(mocktime) - it 'runs with default params' do - allow_apply - pending('a lack of support for functions requires a workaround to be written') - expect_task('peadm::get_peadm_config').always_return({ 'primary_postgresql_host' => 'postgres' }) - expect_out_message.with_params('# Backing up ca and ssl certificates') - # The commands all have a timestamp in them and frankly its proved to hard with bolt spec to work this out - allow_any_command allow_apply + + expect_task('peadm::get_peadm_config').always_return({ + 'params' => { + 'primary_host' => 'primary', + 'primary_postgresql_host' => 'postgres', + } + }) + end + + it 'runs with backup type recovery' do + expect_out_message.with_params('# Backing up ca, certs, code and config for recovery') + expect_out_message.with_params('# Backing up database pe-puppetdb') + + expect_command("/opt/puppetlabs/bin/puppet-backup create --dir=/tmp/pe-backup-1970-01-01T000000Z/recovery --scope=certs,code,config\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/puppetdb/pe-puppetdb.dump.d "sslmode=verify-ca host=postgres user=pe-puppetdb sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb"' + "\n") + expect_command('umask 0077 && cd /tmp && tar -czf /tmp/pe-backup-1970-01-01T000000Z.tar.gz pe-backup-1970-01-01T000000Z && rm -rf /tmp/pe-backup-1970-01-01T000000Z' + "\n") + + expect(run_plan('peadm::backup', default_params)).to be_ok + end + + it 'runs with backup type custom, classifier only' do + expect_task('peadm::backup_classification').with_params({ "directory"=>"/tmp/pe-backup-1970-01-01T000000Z/classifier"}) + expect_out_message.with_params('# Backing up classification') + expect_command('umask 0077 && cd /tmp && tar -czf /tmp/pe-backup-1970-01-01T000000Z.tar.gz pe-backup-1970-01-01T000000Z && rm -rf /tmp/pe-backup-1970-01-01T000000Z' + "\n") + + expect(run_plan('peadm::backup', classifier_only)).to be_ok + end + + it 'runs with backup type custom, all backup params set to true' do + expect_task('peadm::backup_classification').with_params({ "directory"=>"/tmp/pe-backup-1970-01-01T000000Z/classifier"}) + + expect_out_message.with_params('# Backing up classification') expect_out_message.with_params('# Backing up database pe-orchestrator') expect_out_message.with_params('# Backing up database pe-activity') expect_out_message.with_params('# Backing up database pe-rbac') - expect_out_message.with_params('# Backing up classification') - expect_task('peadm::backup_classification') - expect(run_plan('peadm::backup', params)).to be_ok + expect_out_message.with_params('# Backing up database pe-puppetdb') + + expect_command("/opt/puppetlabs/bin/puppet-backup create --dir=/tmp/pe-backup-1970-01-01T000000Z/ca --scope=certs\n") + expect_command("/opt/puppetlabs/bin/puppet-backup create --dir=/tmp/pe-backup-1970-01-01T000000Z/code --scope=code\n") + expect_command("chown pe-postgres /tmp/pe-backup-1970-01-01T000000Z/config") + expect_command("/opt/puppetlabs/bin/puppet-backup create --dir=/tmp/pe-backup-1970-01-01T000000Z/config --scope=config\n") + expect_command("test -f /etc/puppetlabs/console-services/conf.d/secrets/keys.json && cp -rp /etc/puppetlabs/console-services/conf.d/secrets /tmp/pe-backup-1970-01-01T000000Z/rbac/ || echo secret ldap key doesnt exist\n") + expect_command("cp -rp /etc/puppetlabs/orchestration-services/conf.d/secrets /tmp/pe-backup-1970-01-01T000000Z/orchestrator/\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/orchestrator/pe-orchestrator.dump.d "sslmode=verify-ca host=primary user=pe-orchestrator sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-orchestrator"'+"\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/activity/pe-activity.dump.d "sslmode=verify-ca host=primary user=pe-activity sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-activity"'+"\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/rbac/pe-rbac.dump.d "sslmode=verify-ca host=primary user=pe-rbac sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-rbac"'+"\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/puppetdb/pe-puppetdb.dump.d "sslmode=verify-ca host=postgres user=pe-puppetdb sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb"'+"\n") + expect_command('umask 0077 && cd /tmp && tar -czf /tmp/pe-backup-1970-01-01T000000Z.tar.gz pe-backup-1970-01-01T000000Z && rm -rf /tmp/pe-backup-1970-01-01T000000Z' + "\n") + + expect(run_plan('peadm::backup', all_backup_options)).to be_ok end + end From da2662baa880ad681486f0eb409912a6bb405ab5 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Thu, 4 Apr 2024 16:49:00 +0200 Subject: [PATCH 084/110] restore spec WIP --- plans/restore.pp | 4 +- plans/util/configure_external_psql_server.pp | 93 ++++++++++++++++++++ spec/acceptance/peadm_spec/plans/dbttest.pp | 9 ++ spec/plans/restore_spec.rb | 77 ++++++++++++---- 4 files changed, 165 insertions(+), 18 deletions(-) create mode 100644 plans/util/configure_external_psql_server.pp create mode 100644 spec/acceptance/peadm_spec/plans/dbttest.pp diff --git a/plans/restore.pp b/plans/restore.pp index 6ab6014c..df2a8085 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -192,7 +192,7 @@ run_command(@("CMD"/L), $primary_target) test -f ${shellquote($recovery_directory)}/rbac/keys.json \ && cp -rp ${shellquote($recovery_directory)}/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ \ - || echo secret ldap key doesn\'t exist + || echo secret ldap key doesn't exist | CMD # lint:ignore:140chars # IF restoring orchestrator restore the secrets to /etc/puppetlabs/orchestration-services/conf.d/secrets/ @@ -206,7 +206,7 @@ #$database_to_restore.each |Integer $index, Boolean $value | { $restore_databases.each |$name,$database_targets| { - out::message("# Restoring ${name} database") + out::message("# Restoring database pe-${name}") $dbname = "pe-${shellquote($name)}" # Drop pglogical extensions and schema if present diff --git a/plans/util/configure_external_psql_server.pp b/plans/util/configure_external_psql_server.pp new file mode 100644 index 00000000..7ffd1dd5 --- /dev/null +++ b/plans/util/configure_external_psql_server.pp @@ -0,0 +1,93 @@ +plan peadm::util::configure_external_psql_server( + String[1] $db_host, + Boolean $install_pe = false, + String[1] $pe_version = '2023.5.0', + String[1] $pe_platform = 'el-8-x86_64', +) { + $t = get_targets('*') + wait_until_available($t) + + $db_target = get_target($db_host) + parallelize($t + $db_target) |$target| { + $fqdn = run_command('hostname -f', $target) + $target.set_var('certname', $fqdn.first['stdout'].chomp) + } + + $primary_target = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] + $compiler_targets = $t.filter |$n| { $n.vars['role'] == 'compiler' } + + out::message("db_target: ${db_target}") + out::message("db_target certname: ${db_target.peadm::certname()}") + out::message("primary_target: ${primary_target}") + out::message("compiler_targets: ${compiler_targets}") + # fail_plan('testing eop') + run_command("/opt/puppetlabs/bin/puppetserver ca clean --certname ${db_target.peadm::certname()}", $primary_target) + +# lint:ignore:strict_indent + run_task('peadm::mkdir_p_file', $db_target, + path => '/etc/puppetlabs/puppet/puppet.conf', + content => @("HEREDOC"), + [main] + certname = ${db_target.peadm::certname()} + | HEREDOC +# lint:endignore + ) + + run_plan('peadm::util::insert_csr_extension_requests', $db_target, + extension_requests => { + peadm::oid('peadm_role') => 'puppet/puppetdb-database', + peadm::oid('peadm_availability_group') => 'A', + }) + + run_command('systemctl stop pe-puppetdb', $compiler_targets, { _catch_errors => true }) + # run_task('service', $primary_target, { action => 'restart', name => 'pe-puppetdb', _catch_errors => true }) + + if $install_pe { + $pe_conf_data = {} + + $puppetdb_database_temp_config = { + 'puppet_enterprise::profile::database::puppetdb_hosts' => ( + $compiler_targets + $primary_target + ).map |$t| { $t.peadm::certname() }, + } + + $primary_postgresql_pe_conf = peadm::generate_pe_conf({ + 'console_admin_password' => 'not used', + 'puppet_enterprise::puppet_master_host' => $primary_target.peadm::certname(), + 'puppet_enterprise::database_host' => $db_target.peadm::certname(), + } + $puppetdb_database_temp_config + $pe_conf_data) + + # Upload the pe.conf files to the hosts that need them, and ensure correctly + # configured certnames. Right now for these hosts we need to do that by + # staging a puppet.conf file. + + peadm::file_content_upload($primary_postgresql_pe_conf, '/tmp/pe.conf', $db_target) + + $uploaddir = '/tmp' + $pe_tarball_name = "puppet-enterprise-${pe_version}-${pe_platform}.tar.gz" + $pe_tarball_source = "https://s3.amazonaws.com/pe-builds/released/${pe_version}/${pe_tarball_name}" + $upload_tarball_path = "${uploaddir}/${pe_tarball_name}" + + run_task('peadm::download', $db_target, + source => $pe_tarball_source, + path => $upload_tarball_path, + ) + + # Run the PE installer on the puppetdb database hosts + run_task('peadm::pe_install', $db_target, + tarball => $upload_tarball_path, + peconf => '/tmp/pe.conf', + puppet_service_ensure => 'stopped', + ) + } + + run_plan('peadm::subplans::component_install', $db_target, { + primary_host => $primary_target, + avail_group_letter => 'A', + role => 'puppet/puppetdb-database', + }) + + run_task('peadm::puppet_runonce', $compiler_targets) + run_command('systemctl start pe-puppetdb', $compiler_targets, { _catch_errors => true }) + run_task('service', $compiler_targets, { action => 'restart', name => 'pe-puppetserver', _catch_errors => true }) +} diff --git a/spec/acceptance/peadm_spec/plans/dbttest.pp b/spec/acceptance/peadm_spec/plans/dbttest.pp new file mode 100644 index 00000000..844336d1 --- /dev/null +++ b/spec/acceptance/peadm_spec/plans/dbttest.pp @@ -0,0 +1,9 @@ +plan peadm_spec::dbttest( + TargetSpec $targets, +) { + $t = get_target($targets) + out::message('Running dbt test') + out::message("targets: ${t}") + out::message("config: ${t.config}") + out::message("config: ${t.config['ssh']['user']}") +} diff --git a/spec/plans/restore_spec.rb b/spec/plans/restore_spec.rb index adceca3d..43e80dce 100644 --- a/spec/plans/restore_spec.rb +++ b/spec/plans/restore_spec.rb @@ -2,23 +2,68 @@ describe 'peadm::restore' do include BoltSpec::Plans - let(:params) { { 'targets' => 'primary', 'backup_timestamp' => '2022-03-29_16:57:41' } } - - it 'runs with default params' do + let(:recovery_params) { + { + 'targets' => 'primary', + 'input_file' => '/input/file.tar.gz', + 'restore_type' => 'recovery' + } + } + let(:recovery_db_params) { + { + 'targets' => 'primary', + 'input_file' => '/input/file.tar.gz', + 'restore_type' => 'recovery-db' + } + } + let(:cluster) { { 'params' => { 'primary_host' => 'primary', 'primary_postgresql_host' => 'postgres' } } } + before(:each) do allow_apply - pending('a lack of support for functions requires a workaround to be written') - expect_task('peadm::get_peadm_config').always_return({ 'primary_postgresql_host' => 'postgres' }) - expect_out_message.with_params('# Backing up ca and ssl certificates') - # The commands all have a timestamp in them and frankly its proved to hard with bolt spec to work this out + + expect_task('peadm::get_peadm_config').always_return(cluster) + expect_out_message.with_params("cluster: " + cluster.to_s.gsub(/"/,'').gsub(/=>/,' => ')) + expect_download('/input/file/peadm/peadm_config.json') + allow_task('peadm::puppet_runonce') + end + + it 'runs with recovery params' do + # allow_any_command + expect_out_message.with_params('# Restoring database pe-puppetdb') + expect_out_message.with_params('# Restoring ca, certs, code and config for recovery') + + expect_command("umask 0077 && cd /input && tar -xzf /input/file.tar.gz\n") + expect_command("/opt/puppetlabs/bin/puppet-backup restore --scope=certs,code,config --tempdir=/input/file --force /input/file/recovery/pe_backup-*tgz\n") + expect_command("systemctl stop pe-console-services pe-nginx pxp-agent pe-puppetserver pe-orchestration-services puppet pe-puppetdb\n") + expect_command("test -f /input/file/rbac/keys.json && cp -rp /input/file/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ || echo secret ldap key doesn't exist\n") + expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql --tuples-only -d 'pe-puppetdb' -c 'DROP SCHEMA IF EXISTS pglogical CASCADE;'\"\n") + expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql -d 'pe-puppetdb' -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'\"\n") + expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'ALTER USER \\"pe-puppetdb\\" WITH SUPERUSER;\'"' + "\n") + expect_command('/opt/puppetlabs/server/bin/pg_restore -j 4 -d "sslmode=verify-ca host=postgres sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb user=pe-puppetdb" -Fd /input/file/puppetdb/pe-puppetdb.dump.d' + "\n") + expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'ALTER USER \\"pe-puppetdb\\" WITH NOSUPERUSER;\'"' + "\n") + expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'DROP EXTENSION IF EXISTS pglogical CASCADE;\'"' + "\n") + expect_command("/opt/puppetlabs/bin/puppet-infrastructure configure --no-recover\n") + + expect(run_plan('peadm::restore', recovery_params)).to be_ok + end + + it 'runs with recovery-db params' do allow_any_command - expect_out_message.with_params('# Restoring classification') - expect_out_message.with_params('# Backed up current classification to /tmp/classification_backup.json') - expect_out_message.with_params('# Restoring ca and ssl certificates') - expect_out_message.with_params('# Restoring database pe-orchestrator') - expect_out_message.with_params('# Restoring database pe-activity') - expect_out_message.with_params('# Restoring database pe-rbac') - expect_out_message.with_params('# Restoring classification') - expect_task('peadm::backup_classification') - expect(run_plan('peadm::restore', params)).to be_ok + expect_out_message.with_params('# Restoring primary database for recovery') + expect_out_message.with_params('# Restoring database pe-puppetdb') + + # expect_command("umask 0077 && cd /input && tar -xzf /input/file.tar.gz\n") + # expect_command("/opt/puppetlabs/bin/puppet-backup restore --scope=certs,code,config --tempdir=/input/file --force /input/file/recovery/pe_backup-*tgz\n") + # expect_command("systemctl stop pe-console-services pe-nginx pxp-agent pe-puppetserver pe-orchestration-services puppet pe-puppetdb\n") + # expect_command("test -f /input/file/rbac/keys.json && cp -rp /input/file/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ || echo secret ldap key doesn't exist\n") + # expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql --tuples-only -d 'pe-puppetdb' -c 'DROP SCHEMA IF EXISTS pglogical CASCADE;'\"\n") + # expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql -d 'pe-puppetdb' -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'\"\n") + # expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'ALTER USER \\"pe-puppetdb\\" WITH SUPERUSER;\'"' + "\n") + # expect_command('/opt/puppetlabs/server/bin/pg_restore -j 4 -d "sslmode=verify-ca host=postgres sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb user=pe-puppetdb" -Fd /input/file/puppetdb/pe-puppetdb.dump.d' + "\n") + # expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'ALTER USER \\"pe-puppetdb\\" WITH NOSUPERUSER;\'"' + "\n") + # expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'DROP EXTENSION IF EXISTS pglogical CASCADE;\'"' + "\n") + # expect_command("/opt/puppetlabs/bin/puppet-infrastructure configure --no-recover\n") + + expect(run_plan('peadm::restore', recovery_db_params)).to be_ok end + end From e198fa4cb0779b9021a15c5565d1346b7845a090 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Thu, 4 Apr 2024 16:49:46 +0200 Subject: [PATCH 085/110] removing obsolete files --- plans/util/configure_external_psql_server.pp | 93 -------------------- spec/acceptance/peadm_spec/plans/dbttest.pp | 9 -- 2 files changed, 102 deletions(-) delete mode 100644 plans/util/configure_external_psql_server.pp delete mode 100644 spec/acceptance/peadm_spec/plans/dbttest.pp diff --git a/plans/util/configure_external_psql_server.pp b/plans/util/configure_external_psql_server.pp deleted file mode 100644 index 7ffd1dd5..00000000 --- a/plans/util/configure_external_psql_server.pp +++ /dev/null @@ -1,93 +0,0 @@ -plan peadm::util::configure_external_psql_server( - String[1] $db_host, - Boolean $install_pe = false, - String[1] $pe_version = '2023.5.0', - String[1] $pe_platform = 'el-8-x86_64', -) { - $t = get_targets('*') - wait_until_available($t) - - $db_target = get_target($db_host) - parallelize($t + $db_target) |$target| { - $fqdn = run_command('hostname -f', $target) - $target.set_var('certname', $fqdn.first['stdout'].chomp) - } - - $primary_target = $t.filter |$n| { $n.vars['role'] == 'primary' }[0] - $compiler_targets = $t.filter |$n| { $n.vars['role'] == 'compiler' } - - out::message("db_target: ${db_target}") - out::message("db_target certname: ${db_target.peadm::certname()}") - out::message("primary_target: ${primary_target}") - out::message("compiler_targets: ${compiler_targets}") - # fail_plan('testing eop') - run_command("/opt/puppetlabs/bin/puppetserver ca clean --certname ${db_target.peadm::certname()}", $primary_target) - -# lint:ignore:strict_indent - run_task('peadm::mkdir_p_file', $db_target, - path => '/etc/puppetlabs/puppet/puppet.conf', - content => @("HEREDOC"), - [main] - certname = ${db_target.peadm::certname()} - | HEREDOC -# lint:endignore - ) - - run_plan('peadm::util::insert_csr_extension_requests', $db_target, - extension_requests => { - peadm::oid('peadm_role') => 'puppet/puppetdb-database', - peadm::oid('peadm_availability_group') => 'A', - }) - - run_command('systemctl stop pe-puppetdb', $compiler_targets, { _catch_errors => true }) - # run_task('service', $primary_target, { action => 'restart', name => 'pe-puppetdb', _catch_errors => true }) - - if $install_pe { - $pe_conf_data = {} - - $puppetdb_database_temp_config = { - 'puppet_enterprise::profile::database::puppetdb_hosts' => ( - $compiler_targets + $primary_target - ).map |$t| { $t.peadm::certname() }, - } - - $primary_postgresql_pe_conf = peadm::generate_pe_conf({ - 'console_admin_password' => 'not used', - 'puppet_enterprise::puppet_master_host' => $primary_target.peadm::certname(), - 'puppet_enterprise::database_host' => $db_target.peadm::certname(), - } + $puppetdb_database_temp_config + $pe_conf_data) - - # Upload the pe.conf files to the hosts that need them, and ensure correctly - # configured certnames. Right now for these hosts we need to do that by - # staging a puppet.conf file. - - peadm::file_content_upload($primary_postgresql_pe_conf, '/tmp/pe.conf', $db_target) - - $uploaddir = '/tmp' - $pe_tarball_name = "puppet-enterprise-${pe_version}-${pe_platform}.tar.gz" - $pe_tarball_source = "https://s3.amazonaws.com/pe-builds/released/${pe_version}/${pe_tarball_name}" - $upload_tarball_path = "${uploaddir}/${pe_tarball_name}" - - run_task('peadm::download', $db_target, - source => $pe_tarball_source, - path => $upload_tarball_path, - ) - - # Run the PE installer on the puppetdb database hosts - run_task('peadm::pe_install', $db_target, - tarball => $upload_tarball_path, - peconf => '/tmp/pe.conf', - puppet_service_ensure => 'stopped', - ) - } - - run_plan('peadm::subplans::component_install', $db_target, { - primary_host => $primary_target, - avail_group_letter => 'A', - role => 'puppet/puppetdb-database', - }) - - run_task('peadm::puppet_runonce', $compiler_targets) - run_command('systemctl start pe-puppetdb', $compiler_targets, { _catch_errors => true }) - run_task('service', $compiler_targets, { action => 'restart', name => 'pe-puppetserver', _catch_errors => true }) -} diff --git a/spec/acceptance/peadm_spec/plans/dbttest.pp b/spec/acceptance/peadm_spec/plans/dbttest.pp deleted file mode 100644 index 844336d1..00000000 --- a/spec/acceptance/peadm_spec/plans/dbttest.pp +++ /dev/null @@ -1,9 +0,0 @@ -plan peadm_spec::dbttest( - TargetSpec $targets, -) { - $t = get_target($targets) - out::message('Running dbt test') - out::message("targets: ${t}") - out::message("config: ${t.config}") - out::message("config: ${t.config['ssh']['user']}") -} From eec3fb743d346f17635d41388799d907da0a1b48 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 5 Apr 2024 13:26:01 +0200 Subject: [PATCH 086/110] add restore spec --- plans/restore.pp | 34 +++++++++++++---------- spec/plans/restore_spec.rb | 55 ++++++++++++++++++++++++++------------ 2 files changed, 58 insertions(+), 31 deletions(-) diff --git a/plans/restore.pp b/plans/restore.pp index df2a8085..ce08e9dc 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -65,7 +65,7 @@ 'recovery' => peadm::recovery_opts_default(), 'recovery-db' => { 'puppetdb' => true, }, 'migration' => peadm::migration_opts_default(), - 'custom' => peadm::migration_opts_all() + $restore, + 'custom' => peadm::recovery_opts_all() + $restore, } $primary_target = peadm::get_targets(getvar('cluster.params.primary_host'), 1) @@ -104,19 +104,25 @@ } if getvar('recovery_opts.classifier') { - out::message('# Restoring classification') - run_task('peadm::backup_classification', $primary_target, - directory => $recovery_directory - ) - - run_task('peadm::transform_classification_groups', $primary_target, - source_directory => "${recovery_directory}/classifier", - working_directory => $recovery_directory - ) - - run_task('peadm::restore_classification', $primary_target, - classification_file => "${recovery_directory}/transformed_classification.json", - ) + if $restore_type == 'migration' { + out::message('# Migrating classification') + run_task('peadm::backup_classification', $primary_target, + directory => $recovery_directory + ) + + run_task('peadm::transform_classification_groups', $primary_target, + source_directory => "${recovery_directory}/classifier", + working_directory => $recovery_directory + ) + + run_task('peadm::restore_classification', $primary_target, + classification_file => "${recovery_directory}/transformed_classification.json", + ) + } else { + run_task('peadm::restore_classification', $primary_target, + classification_file => "${recovery_directory}/classifier/classification_backup.json", + ) + } } if $restore_type == 'recovery' { diff --git a/spec/plans/restore_spec.rb b/spec/plans/restore_spec.rb index 43e80dce..ac079d8c 100644 --- a/spec/plans/restore_spec.rb +++ b/spec/plans/restore_spec.rb @@ -2,32 +2,54 @@ describe 'peadm::restore' do include BoltSpec::Plans + + backup_dir = '/input/file' + backup_tarball = "#{backup_dir}.tar.gz" + let(:recovery_params) { { 'targets' => 'primary', - 'input_file' => '/input/file.tar.gz', + 'input_file' => backup_tarball, 'restore_type' => 'recovery' } } let(:recovery_db_params) { { 'targets' => 'primary', - 'input_file' => '/input/file.tar.gz', + 'input_file' => backup_tarball, 'restore_type' => 'recovery-db' } } + let(:classifier_only_params) { + { + 'targets' => 'primary', + 'input_file' => backup_tarball, + 'restore_type' => 'custom', + 'restore' => { + 'activity' => false, + 'ca' => false, + 'classifier' => true, + 'code' => false, + 'config' => false, + 'orchestrator' => false, + 'puppetdb' => false, + 'rbac' => false, + } + } + } + let(:cluster) { { 'params' => { 'primary_host' => 'primary', 'primary_postgresql_host' => 'postgres' } } } before(:each) do allow_apply expect_task('peadm::get_peadm_config').always_return(cluster) expect_out_message.with_params("cluster: " + cluster.to_s.gsub(/"/,'').gsub(/=>/,' => ')) - expect_download('/input/file/peadm/peadm_config.json') + expect_out_message.with_params('# Restoring ldap secret key if it exists') + expect_download("#{backup_dir}/peadm/peadm_config.json") allow_task('peadm::puppet_runonce') end it 'runs with recovery params' do - # allow_any_command expect_out_message.with_params('# Restoring database pe-puppetdb') expect_out_message.with_params('# Restoring ca, certs, code and config for recovery') @@ -35,7 +57,7 @@ expect_command("/opt/puppetlabs/bin/puppet-backup restore --scope=certs,code,config --tempdir=/input/file --force /input/file/recovery/pe_backup-*tgz\n") expect_command("systemctl stop pe-console-services pe-nginx pxp-agent pe-puppetserver pe-orchestration-services puppet pe-puppetdb\n") expect_command("test -f /input/file/rbac/keys.json && cp -rp /input/file/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ || echo secret ldap key doesn't exist\n") - expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql --tuples-only -d 'pe-puppetdb' -c 'DROP SCHEMA IF EXISTS pglogical CASCADE;'\"\n") + expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql --tuples-only -d 'pe-puppetdb' -c 'DROP SCHEMA IF EXISTS pglogical CASCADE;'\"\n").be_called_times(2) expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql -d 'pe-puppetdb' -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'\"\n") expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'ALTER USER \\"pe-puppetdb\\" WITH SUPERUSER;\'"' + "\n") expect_command('/opt/puppetlabs/server/bin/pg_restore -j 4 -d "sslmode=verify-ca host=postgres sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb user=pe-puppetdb" -Fd /input/file/puppetdb/pe-puppetdb.dump.d' + "\n") @@ -48,22 +70,21 @@ it 'runs with recovery-db params' do allow_any_command + expect_out_message.with_params('# Restoring primary database for recovery') expect_out_message.with_params('# Restoring database pe-puppetdb') - # expect_command("umask 0077 && cd /input && tar -xzf /input/file.tar.gz\n") - # expect_command("/opt/puppetlabs/bin/puppet-backup restore --scope=certs,code,config --tempdir=/input/file --force /input/file/recovery/pe_backup-*tgz\n") - # expect_command("systemctl stop pe-console-services pe-nginx pxp-agent pe-puppetserver pe-orchestration-services puppet pe-puppetdb\n") - # expect_command("test -f /input/file/rbac/keys.json && cp -rp /input/file/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ || echo secret ldap key doesn't exist\n") - # expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql --tuples-only -d 'pe-puppetdb' -c 'DROP SCHEMA IF EXISTS pglogical CASCADE;'\"\n") - # expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql -d 'pe-puppetdb' -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'\"\n") - # expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'ALTER USER \\"pe-puppetdb\\" WITH SUPERUSER;\'"' + "\n") - # expect_command('/opt/puppetlabs/server/bin/pg_restore -j 4 -d "sslmode=verify-ca host=postgres sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb user=pe-puppetdb" -Fd /input/file/puppetdb/pe-puppetdb.dump.d' + "\n") - # expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'ALTER USER \\"pe-puppetdb\\" WITH NOSUPERUSER;\'"' + "\n") - # expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'DROP EXTENSION IF EXISTS pglogical CASCADE;\'"' + "\n") - # expect_command("/opt/puppetlabs/bin/puppet-infrastructure configure --no-recover\n") - expect(run_plan('peadm::restore', recovery_db_params)).to be_ok end + it 'runs with classifier-only params' do + allow_any_command + + expect_task('peadm::restore_classification').with_params({ + "classification_file" => "#{backup_dir}/classifier/classification_backup.json" + }) + + expect(run_plan('peadm::restore', classifier_only_params)).to be_ok + end + end From 6daff4f9924defae2171a050022443214863f181 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 5 Apr 2024 16:27:29 +0200 Subject: [PATCH 087/110] add peadm_config fixtures to test broken cluster restore --- plans/restore.pp | 6 ++---- spec/fixtures/peadm_config.json | 3 +++ spec/plans/restore_spec.rb | 32 +++++++++++++++++++++++++++----- 3 files changed, 32 insertions(+), 9 deletions(-) create mode 100644 spec/fixtures/peadm_config.json diff --git a/plans/restore.pp b/plans/restore.pp index ce08e9dc..d0f5811a 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -34,13 +34,11 @@ # try to load the cluster configuration by running peadm::get_peadm_config, but allow for errors to happen $_cluster = run_task('peadm::get_peadm_config', $targets, { '_catch_errors' => true }).first.value - $result = download_file("${recovery_directory}/peadm/peadm_config.json", 'peadm_config.json', $targets).first.value - $cluster_backup = loadjson(getvar('result.path')) - if $_cluster == undef or getvar('_cluster.params') == undef { # failed to get cluster config, load from backup out::message('Failed to get cluster configuration, loading from backup...') - $cluster = $cluster_backup + $result = download_file("${recovery_directory}/peadm/peadm_config.json", 'peadm_config.json', $targets).first.value + $cluster = loadjson(getvar('result.path')) out::message('Cluster configuration loaded from backup') } else { $cluster = $_cluster diff --git a/spec/fixtures/peadm_config.json b/spec/fixtures/peadm_config.json new file mode 100644 index 00000000..cbd5db76 --- /dev/null +++ b/spec/fixtures/peadm_config.json @@ -0,0 +1,3 @@ +{ + "params": { "primary_host": "primary", "primary_postgresql_host": "postgres" } +} diff --git a/spec/plans/restore_spec.rb b/spec/plans/restore_spec.rb index ac079d8c..9f3805ec 100644 --- a/spec/plans/restore_spec.rb +++ b/spec/plans/restore_spec.rb @@ -39,17 +39,21 @@ } let(:cluster) { { 'params' => { 'primary_host' => 'primary', 'primary_postgresql_host' => 'postgres' } } } + before(:each) do allow_apply - expect_task('peadm::get_peadm_config').always_return(cluster) expect_out_message.with_params("cluster: " + cluster.to_s.gsub(/"/,'').gsub(/=>/,' => ')) expect_out_message.with_params('# Restoring ldap secret key if it exists') - expect_download("#{backup_dir}/peadm/peadm_config.json") allow_task('peadm::puppet_runonce') end - it 'runs with recovery params' do + # only run for tests that have the :valid_cluster tag + before(:each, :valid_cluster => true) do + expect_task('peadm::get_peadm_config').always_return(cluster) + end + + it 'runs with recovery params', :valid_cluster => true do expect_out_message.with_params('# Restoring database pe-puppetdb') expect_out_message.with_params('# Restoring ca, certs, code and config for recovery') @@ -68,7 +72,7 @@ expect(run_plan('peadm::restore', recovery_params)).to be_ok end - it 'runs with recovery-db params' do + it 'runs with recovery-db params', :valid_cluster => true do allow_any_command expect_out_message.with_params('# Restoring primary database for recovery') @@ -77,7 +81,7 @@ expect(run_plan('peadm::restore', recovery_db_params)).to be_ok end - it 'runs with classifier-only params' do + it 'runs with classifier-only params', :valid_cluster => true do allow_any_command expect_task('peadm::restore_classification').with_params({ @@ -87,4 +91,22 @@ expect(run_plan('peadm::restore', classifier_only_params)).to be_ok end + it 'runs with recovery params, no valid cluster', :valid_cluster => false do + allow_any_command + + # simulate a failure to get the cluster configuration + expect_task('peadm::get_peadm_config').always_return({}) + expect_out_message.with_params("Failed to get cluster configuration, loading from backup...") + + # download mocked to return the path to the file fixtures/peadm_config.json + expect_download("#{backup_dir}/peadm/peadm_config.json").return do |targets, source, destination, params| + results = targets.map do |target| + Bolt::Result.new(target, value: { 'path' => File.expand_path(File.join(fixtures, 'peadm_config.json'))}) + end + + Bolt::ResultSet.new(results) + end + expect(run_plan('peadm::restore', recovery_params)).to be_ok + end + end From 83576510873e5fdd114a730319b25196525bf368 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 5 Apr 2024 16:48:32 +0200 Subject: [PATCH 088/110] Add require 'spec_helper' to sanitize_pg_pe_conf_spec.rb --- spec/plans/util/sanitize_pg_pe_conf_spec.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/spec/plans/util/sanitize_pg_pe_conf_spec.rb b/spec/plans/util/sanitize_pg_pe_conf_spec.rb index 25384656..38b5521e 100644 --- a/spec/plans/util/sanitize_pg_pe_conf_spec.rb +++ b/spec/plans/util/sanitize_pg_pe_conf_spec.rb @@ -1,4 +1,4 @@ -# spec/spec_helper.rb +require 'spec_helper' describe 'peadm::util::sanitize_pg_pe_conf ' do # Include the BoltSpec library functions @@ -9,6 +9,7 @@ end it 'Runs' do + allow_any_out_message # 1) peadm::util::sanitize_pg_pe_conf Runs # Failure/Error: expect(run_plan('peadm::util::sanitize_pg_pe_conf', 'targets' => 'foo,bar', 'primary_host' => 'pe-server-d8b317-0.us-west1-a.c.davidsand.internal')).to be_ok # expected `#>, @status="failure">.ok?` to be truthy, got false From 1dd6d77ef42532961af082aedb950d7504543d75 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Fri, 5 Apr 2024 17:22:23 +0200 Subject: [PATCH 089/110] Update documentation --- documentation/backup_restore.md | 74 ++++++++++++++++++++++++++++----- plans/util/init_db_server.pp | 1 - 2 files changed, 64 insertions(+), 11 deletions(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 7fc19b89..0e815579 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -1,5 +1,15 @@ # Backup and restore Puppet Enterprise (PE) +- [Backup and restore Puppet Enterprise (PE)](#backup-and-restore-puppet-enterprise-pe) + - [Introduction to PEADM backup and restore](#introduction-to-peadm-backup-and-restore) + - [Using `recovery` backup and restore](#using-recovery-backup-and-restore) + - [Using `custom` backup and restore](#using-custom-backup-and-restore) + - [What exactly is backed up and restored?](#what-exactly-is-backed-up-and-restored) + - [Recovering a broken primary server](#recovering-a-broken-primary-server) + - [Recovering a broken external primary database server in an Extra Large (XL) installation](#recovering-a-broken-external-primary-database-server-in-an-extra-large-xl-installation) + +## Introduction to PEADM backup and restore + If your PE installation is managed by PEADM, you can back up and restore PE using this process: 1. Use the `peadm::backup` plan to create a backup of your primary server. 2. Use the `peadm::restore` plan to restore PE from a `peadm::backup`. @@ -12,6 +22,8 @@ When running the backup and restore plans, you can define the `backup_type` and If no type is specified, the default is `recovery`. +(When running the restore plan, an additional restore type `recovery-db` is available to recover a failed primary database server (see [Recovering a broken external primary database server in an Extra Large (XL) installation](#recovering-a-broken-external-primary-database-server-in-an-extra-large-xl-installation)).) + When backing up or restoring PE, you must use the `--targets` option to specify the hostname (FQDN) of your primary server. The backup file created is named `pe-backup-YYYY-MM-DDTHHMMSSZ.tar.gz` and placed by default in `/tmp`. To specify a different location for the backup file, you can define the `output_directory` parameter. @@ -105,17 +117,59 @@ bolt plan run peadm::restore --targets my.primary.vm --params @params.json The following table shows the items you can specify and indicates what is included in `recovery`: -| Data or service | Explanation | Used in `recovery` | -| ------------------| -------------------------------------------------------------------------------------------------------- | ------------------ | -| `activity ` | Activity database | | -| `ca ` | CA and ssl certificates | ✅ | -| `classifier` | Classifier database. Restore merges user-defined node groups rather than overwriting system node groups. | | -| `code` | Code directory | ✅ | -| `config` | Configuration files and databases (databases are restored literally) | ✅ | -| `orchestrator ` | Orchestrator database | | -| `puppetdb` | PuppetDB database (including support for XL where puppetdb is running on an external db server) | ✅ | -| `rbac` | RBAC database | | +| Data or service | Explanation | Used in `recovery` | +| --------------- | -------------------------------------------------------------------------------------------------------- | ------------------ | +| `activity ` | Activity database | | +| `ca ` | CA and ssl certificates | ✅ | +| `classifier` | Classifier database. Restore merges user-defined node groups rather than overwriting system node groups. | | +| `code` | Code directory | ✅ | +| `config` | Configuration files and databases (databases are restored literally) | ✅ | +| `orchestrator ` | Orchestrator database | | +| `puppetdb` | PuppetDB database (including support for XL where puppetdb is running on an external db server) | ✅ | +| `rbac` | RBAC database | | **Note**: The PEADM backup and restore plans utilize the `puppet-backup` tool for backing up and restoring `ca`, `code` and `config`. For `config`, the data backed up includes the `activity`, `classifier`, `orchestrator`, and `rbac` databases. **Note:** The output for the `peadm::backup` plan differs from the output that is returned when you manually run the [`puppet-backup create` command](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure). + +## Recovering a broken primary server + +When your primary server is broken, you might not be able to use the `recovery` restore directly because the primary server is not in an operational state. In this case, follow the steps below to restore your primary server: +1. Make sure you have a `recovery` backup of your primary server. +1. Uninstall and reinstall Puppet Enterprise on the broken primary server. Make sure you are installing the same PE version as your current primary server was running. Optionally, you can use the `peadm::reinstall_pe` task as follows: + ``` + bolt task run peadm::reinstall_pe --targets my.primary.vm uninstall=true version=2023.5.0 + ``` +1. Perform a `recovery` restore of your primary server using the backup. For example: + ``` + bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery + ``` + +## Recovering a broken external primary database server in an Extra Large (XL) installation + +When your primary database server is broken, you might not be able to use the `recovery` restore directly because the puppetdb database will not be operational. In this case, follow the steps below to restore your primary database: +1. Make sure you have a `recovery` backup of your primary server. +1. Reinstall Puppet Enterprise on the broken database server and reconfigure and re-sign its certificate. Make sure you are installing the same PE version as your current primary server was running. +To do this, use the plan `peadm::util::init_db_server` as follows: + ``` + bolt plan run peadm::util::init_db_server db_host=my.primary_db.vm pe_version=2023.5.0 install_pe=true + ``` + + This plan will perform the following: + + 1. Clean the current certificate for the database server from the primary server. + 1. Request a new certificate for the database server with the right extensions (peadm_role = puppet/puppetdb-database, peadm_availability_group=A). + 1. Stop the puppetdb service on the compilers. + 1. Prepare a `pe.conf` file on the database server for database installation + 1. Install PE on the database server using the generated `pe.conf` file. + 1. Configure the database as the primary puppetdb database in the XL installation. + 1. Run puppet on the compilers to allow puppetdb on the compilers to be reconfigured with the new primary database server. + 1. Start the puppetdb service on the compilers. + 1. Restart the puppetserver service on the compilers. + +1. Perform a `recovery-db` restore of your primary database server using the backup. For example: + ``` + bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery-db + ``` + **Note**: You should use the `restore_type=recovery-db` parameter to recover the database server. + **Note**: You should specify the primary host (not the primary database host) as the target for the restore plan. diff --git a/plans/util/init_db_server.pp b/plans/util/init_db_server.pp index 8752803e..0b5cf2be 100644 --- a/plans/util/init_db_server.pp +++ b/plans/util/init_db_server.pp @@ -20,7 +20,6 @@ out::message("db_target certname: ${db_target.peadm::certname()}") out::message("primary_target: ${primary_target}") out::message("compiler_targets: ${compiler_targets}") - # fail_plan('testing eop') run_command("/opt/puppetlabs/bin/puppetserver ca clean --certname ${db_target.peadm::certname()}", $primary_target) # lint:ignore:strict_indent From e49d6b6c430aa183adc0864572634ddb1c20a62e Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Sat, 6 Apr 2024 20:42:40 +0200 Subject: [PATCH 090/110] fix rubocop checks --- .vscode/extensions.json | 6 --- bolt-project.yaml | 2 +- spec/plans/backup_spec.rb | 65 +++++++++++++----------- spec/plans/restore_spec.rb | 65 +++++++++++++----------- tasks/get_peadm_config.rb | 2 +- tasks/transform_classification_groups.rb | 22 ++++---- 6 files changed, 82 insertions(+), 80 deletions(-) delete mode 100644 .vscode/extensions.json mode change 100644 => 100755 tasks/transform_classification_groups.rb diff --git a/.vscode/extensions.json b/.vscode/extensions.json deleted file mode 100644 index 6b391843..00000000 --- a/.vscode/extensions.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "recommendations": [ - "puppet.puppet-vscode", - "Shopify.ruby-lsp", - ] -} \ No newline at end of file diff --git a/bolt-project.yaml b/bolt-project.yaml index 306fd6ef..e1a6d593 100644 --- a/bolt-project.yaml +++ b/bolt-project.yaml @@ -6,4 +6,4 @@ modules: - puppet/format - name: node_manager git: https://github.com/WhatsARanjit/puppet-node_manager - ref: master + ref: 86cb48f27a4a0ea212b33f86775d84a374a189b7 diff --git a/spec/plans/backup_spec.rb b/spec/plans/backup_spec.rb index 113e269b..3c0a8e36 100644 --- a/spec/plans/backup_spec.rb +++ b/spec/plans/backup_spec.rb @@ -1,13 +1,17 @@ +# frozen_string_literal: true + +# rubocop:disable Layout/LineLength + require 'spec_helper' describe 'peadm::backup' do include BoltSpec::Plans let(:default_params) { { 'targets' => 'primary', 'backup_type' => 'recovery' } } - let(:classifier_only) { - { - 'targets' => 'primary', - 'backup_type' => 'custom', - 'backup' => { + let(:classifier_only) do + { + 'targets' => 'primary', + 'backup_type' => 'custom', + 'backup' => { 'activity' => false, 'ca' => false, 'classifier' => true, @@ -16,16 +20,16 @@ 'orchestrator' => false, 'puppetdb' => false, 'rbac' => false, - } - } - } - let(:all_backup_options) { - { - 'targets' => 'primary', - 'backup_type' => 'custom', - 'backup' => {} # set all to true - } - } + } + } + end + let(:all_backup_options) do + { + 'targets' => 'primary', + 'backup_type' => 'custom', + 'backup' => {} # set all to true + } + end before(:each) do # define a zero timestamp @@ -36,12 +40,12 @@ allow_apply - expect_task('peadm::get_peadm_config').always_return({ - 'params' => { - 'primary_host' => 'primary', - 'primary_postgresql_host' => 'postgres', - } - }) + expect_task('peadm::get_peadm_config').always_return({ + 'params' => { + 'primary_host' => 'primary', + 'primary_postgresql_host' => 'postgres', + } + }) end it 'runs with backup type recovery' do @@ -56,15 +60,15 @@ end it 'runs with backup type custom, classifier only' do - expect_task('peadm::backup_classification').with_params({ "directory"=>"/tmp/pe-backup-1970-01-01T000000Z/classifier"}) + expect_task('peadm::backup_classification').with_params({ 'directory' => '/tmp/pe-backup-1970-01-01T000000Z/classifier' }) expect_out_message.with_params('# Backing up classification') expect_command('umask 0077 && cd /tmp && tar -czf /tmp/pe-backup-1970-01-01T000000Z.tar.gz pe-backup-1970-01-01T000000Z && rm -rf /tmp/pe-backup-1970-01-01T000000Z' + "\n") - + expect(run_plan('peadm::backup', classifier_only)).to be_ok end it 'runs with backup type custom, all backup params set to true' do - expect_task('peadm::backup_classification').with_params({ "directory"=>"/tmp/pe-backup-1970-01-01T000000Z/classifier"}) + expect_task('peadm::backup_classification').with_params({ 'directory' => '/tmp/pe-backup-1970-01-01T000000Z/classifier' }) expect_out_message.with_params('# Backing up classification') expect_out_message.with_params('# Backing up database pe-orchestrator') @@ -74,17 +78,16 @@ expect_command("/opt/puppetlabs/bin/puppet-backup create --dir=/tmp/pe-backup-1970-01-01T000000Z/ca --scope=certs\n") expect_command("/opt/puppetlabs/bin/puppet-backup create --dir=/tmp/pe-backup-1970-01-01T000000Z/code --scope=code\n") - expect_command("chown pe-postgres /tmp/pe-backup-1970-01-01T000000Z/config") + expect_command('chown pe-postgres /tmp/pe-backup-1970-01-01T000000Z/config') expect_command("/opt/puppetlabs/bin/puppet-backup create --dir=/tmp/pe-backup-1970-01-01T000000Z/config --scope=config\n") expect_command("test -f /etc/puppetlabs/console-services/conf.d/secrets/keys.json && cp -rp /etc/puppetlabs/console-services/conf.d/secrets /tmp/pe-backup-1970-01-01T000000Z/rbac/ || echo secret ldap key doesnt exist\n") expect_command("cp -rp /etc/puppetlabs/orchestration-services/conf.d/secrets /tmp/pe-backup-1970-01-01T000000Z/orchestrator/\n") - expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/orchestrator/pe-orchestrator.dump.d "sslmode=verify-ca host=primary user=pe-orchestrator sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-orchestrator"'+"\n") - expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/activity/pe-activity.dump.d "sslmode=verify-ca host=primary user=pe-activity sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-activity"'+"\n") - expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/rbac/pe-rbac.dump.d "sslmode=verify-ca host=primary user=pe-rbac sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-rbac"'+"\n") - expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/puppetdb/pe-puppetdb.dump.d "sslmode=verify-ca host=postgres user=pe-puppetdb sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb"'+"\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/orchestrator/pe-orchestrator.dump.d "sslmode=verify-ca host=primary user=pe-orchestrator sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-orchestrator"' + "\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/activity/pe-activity.dump.d "sslmode=verify-ca host=primary user=pe-activity sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-activity"' + "\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/rbac/pe-rbac.dump.d "sslmode=verify-ca host=primary user=pe-rbac sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-rbac"' + "\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/puppetdb/pe-puppetdb.dump.d "sslmode=verify-ca host=postgres user=pe-puppetdb sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb"' + "\n") expect_command('umask 0077 && cd /tmp && tar -czf /tmp/pe-backup-1970-01-01T000000Z.tar.gz pe-backup-1970-01-01T000000Z && rm -rf /tmp/pe-backup-1970-01-01T000000Z' + "\n") - + expect(run_plan('peadm::backup', all_backup_options)).to be_ok end - end diff --git a/spec/plans/restore_spec.rb b/spec/plans/restore_spec.rb index 9f3805ec..5df0c5f3 100644 --- a/spec/plans/restore_spec.rb +++ b/spec/plans/restore_spec.rb @@ -1,31 +1,35 @@ +# frozen_string_literal: true + +# rubocop:disable Layout/LineLength + require 'spec_helper' describe 'peadm::restore' do include BoltSpec::Plans - - backup_dir = '/input/file' + + backup_dir = '/input/file' backup_tarball = "#{backup_dir}.tar.gz" - let(:recovery_params) { - { - 'targets' => 'primary', + let(:recovery_params) do + { + 'targets' => 'primary', 'input_file' => backup_tarball, 'restore_type' => 'recovery' } - } - let(:recovery_db_params) { - { - 'targets' => 'primary', + end + let(:recovery_db_params) do + { + 'targets' => 'primary', 'input_file' => backup_tarball, 'restore_type' => 'recovery-db' - } - } - let(:classifier_only_params) { - { - 'targets' => 'primary', + } + end + let(:classifier_only_params) do + { + 'targets' => 'primary', 'input_file' => backup_tarball, - 'restore_type' => 'custom', - 'restore' => { + 'restore_type' => 'custom', + 'restore' => { 'activity' => false, 'ca' => false, 'classifier' => true, @@ -34,26 +38,26 @@ 'orchestrator' => false, 'puppetdb' => false, 'rbac' => false, - } - } - } + } + } + end let(:cluster) { { 'params' => { 'primary_host' => 'primary', 'primary_postgresql_host' => 'postgres' } } } before(:each) do allow_apply - expect_out_message.with_params("cluster: " + cluster.to_s.gsub(/"/,'').gsub(/=>/,' => ')) + expect_out_message.with_params('cluster: ' + cluster.to_s.delete('"').gsub(%r{=>}, ' => ')) expect_out_message.with_params('# Restoring ldap secret key if it exists') allow_task('peadm::puppet_runonce') end # only run for tests that have the :valid_cluster tag - before(:each, :valid_cluster => true) do + before(:each, valid_cluster: true) do expect_task('peadm::get_peadm_config').always_return(cluster) end - it 'runs with recovery params', :valid_cluster => true do + it 'runs with recovery params', valid_cluster: true do expect_out_message.with_params('# Restoring database pe-puppetdb') expect_out_message.with_params('# Restoring ca, certs, code and config for recovery') @@ -72,7 +76,7 @@ expect(run_plan('peadm::restore', recovery_params)).to be_ok end - it 'runs with recovery-db params', :valid_cluster => true do + it 'runs with recovery-db params', valid_cluster: true do allow_any_command expect_out_message.with_params('# Restoring primary database for recovery') @@ -81,32 +85,31 @@ expect(run_plan('peadm::restore', recovery_db_params)).to be_ok end - it 'runs with classifier-only params', :valid_cluster => true do + it 'runs with classifier-only params', valid_cluster: true do allow_any_command expect_task('peadm::restore_classification').with_params({ - "classification_file" => "#{backup_dir}/classifier/classification_backup.json" - }) + 'classification_file' => "#{backup_dir}/classifier/classification_backup.json" + }) expect(run_plan('peadm::restore', classifier_only_params)).to be_ok end - it 'runs with recovery params, no valid cluster', :valid_cluster => false do + it 'runs with recovery params, no valid cluster', valid_cluster: false do allow_any_command # simulate a failure to get the cluster configuration expect_task('peadm::get_peadm_config').always_return({}) - expect_out_message.with_params("Failed to get cluster configuration, loading from backup...") + expect_out_message.with_params('Failed to get cluster configuration, loading from backup...') # download mocked to return the path to the file fixtures/peadm_config.json - expect_download("#{backup_dir}/peadm/peadm_config.json").return do |targets, source, destination, params| + expect_download("#{backup_dir}/peadm/peadm_config.json").return do |targets, _source, _destination, _params| results = targets.map do |target| - Bolt::Result.new(target, value: { 'path' => File.expand_path(File.join(fixtures, 'peadm_config.json'))}) + Bolt::Result.new(target, value: { 'path' => File.expand_path(File.join(fixtures, 'peadm_config.json')) }) end Bolt::ResultSet.new(results) end expect(run_plan('peadm::restore', recovery_params)).to be_ok end - end diff --git a/tasks/get_peadm_config.rb b/tasks/get_peadm_config.rb index 9bf0993f..30d8ad21 100755 --- a/tasks/get_peadm_config.rb +++ b/tasks/get_peadm_config.rb @@ -16,7 +16,7 @@ def execute! if replica_group puts config.to_json else - puts ({ 'error' => 'This is not a peadm-compatible cluster. Use peadm::convert first.' }).to_json + puts({ 'error' => 'This is not a peadm-compatible cluster. Use peadm::convert first.' }).to_json end end diff --git a/tasks/transform_classification_groups.rb b/tasks/transform_classification_groups.rb old mode 100644 new mode 100755 index ab025a40..eeb031e4 --- a/tasks/transform_classification_groups.rb +++ b/tasks/transform_classification_groups.rb @@ -1,6 +1,8 @@ #!/opt/puppetlabs/puppet/bin/ruby # frozen_string_literal: true +# rubocop:disable Naming/VariableName + # This script takes two classification outputs from source and target puppet infrastructure and # takes the user definitions from the source and adds them to the infrastructure definitions of the # target. This allows the ability to restore a backup of user node definitions. @@ -15,10 +17,10 @@ # Function to remove subgroups def removesubgroups(data_rsg, id_rsg) - groups = data_rsg.select { |x| x["parent"] == id_rsg } + groups = data_rsg.select { |x| x['parent'] == id_rsg } groups.each do |group| - subid = group["id"] - data_rsg.reject! { |x| x["id"] == subid } + subid = group['id'] + data_rsg.reject! { |x| x['id'] == subid } data_rsg = removesubgroups(data_rsg, subid) end data_rsg @@ -26,10 +28,10 @@ def removesubgroups(data_rsg, id_rsg) # Function to add subgroups def addsubgroups(data_asg, id_asg, peinf_asg) - groups = data_asg.select { |x| x["parent"] == id_asg } + groups = data_asg.select { |x| x['parent'] == id_asg } peinf_asg += groups groups.each do |group| - subid = group["id"] + subid = group['id'] peinf_asg = addsubgroups(data_asg, subid, peinf_asg) end peinf_asg @@ -42,16 +44,16 @@ def addsubgroups(data_asg, id_asg, peinf_asg) data_DR = JSON.parse(File.read(target_classification_file)) # Find the infrastructure group and its ID -peinf = data.select { |x| x["name"] == "PE Infrastructure" } -group_id = peinf[0]["id"] +peinf = data.select { |x| x['name'] == 'PE Infrastructure' } +group_id = peinf[0]['id'] # Remove this group from the list and recursively remove all subgroups -data.reject! { |x| x["id"] == group_id } +data.reject! { |x| x['id'] == group_id } data = removesubgroups(data, group_id) # Find the DR infrastructure group and its ID -peinf_DR = data_DR.select { |x| x["name"] == "PE Infrastructure" } -id_DR = peinf_DR[0]["id"] +peinf_DR = data_DR.select { |x| x['name'] == 'PE Infrastructure' } +id_DR = peinf_DR[0]['id'] # Recursively go through inf groups to get the full tree peinf_DR = addsubgroups(data_DR, id_DR, peinf_DR) From 57f60b75fc256d4b70b5328f59f6105aeb7aafa8 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Mon, 8 Apr 2024 15:54:12 +0100 Subject: [PATCH 091/110] Adding couple more spec cases --- plans/backup.pp | 4 ++-- spec/plans/backup_spec.rb | 22 ++++++++++++++++++++++ spec/plans/restore_spec.rb | 9 +++++++++ 3 files changed, 33 insertions(+), 2 deletions(-) diff --git a/plans/backup.pp b/plans/backup.pp index df6b52c4..80835eb0 100644 --- a/plans/backup.pp +++ b/plans/backup.pp @@ -1,11 +1,11 @@ # @summary Backup puppet primary configuration # -# @param targets These are a list of the primary puppetservers from one or multiple puppet clusters +# @param targets This should be the primary puppetserver for the puppet cluster # @param backup_type Currently, the recovery and custom backup types are supported # @param backup A hash of custom backup options, see the peadm::recovery_opts_default() function for the default values # @param output_directory The directory to place the backup in # @example -# bolt plan run peadm::backup -t primary1.example.com,primary2.example.com +# bolt plan run peadm::backup -t primary1.example.com # plan peadm::backup ( # This plan should be run on the primary server diff --git a/spec/plans/backup_spec.rb b/spec/plans/backup_spec.rb index 3c0a8e36..d6b2ff75 100644 --- a/spec/plans/backup_spec.rb +++ b/spec/plans/backup_spec.rb @@ -59,6 +59,28 @@ expect(run_plan('peadm::backup', default_params)).to be_ok end + it 'runs with backup type recovery by default' do + expect_out_message.with_params('# Backing up ca, certs, code and config for recovery') + expect_out_message.with_params('# Backing up database pe-puppetdb') + + expect_command("/opt/puppetlabs/bin/puppet-backup create --dir=/tmp/pe-backup-1970-01-01T000000Z/recovery --scope=certs,code,config\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /tmp/pe-backup-1970-01-01T000000Z/puppetdb/pe-puppetdb.dump.d "sslmode=verify-ca host=postgres user=pe-puppetdb sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb"' + "\n") + expect_command('umask 0077 && cd /tmp && tar -czf /tmp/pe-backup-1970-01-01T000000Z.tar.gz pe-backup-1970-01-01T000000Z && rm -rf /tmp/pe-backup-1970-01-01T000000Z' + "\n") + + expect(run_plan('peadm::backup', { 'targets' => 'primary' })).to be_ok + end + + it 'runs with backup and defined output folder' do + expect_out_message.with_params('# Backing up ca, certs, code and config for recovery') + expect_out_message.with_params('# Backing up database pe-puppetdb') + + expect_command("/opt/puppetlabs/bin/puppet-backup create --dir=/user/home/folder/pe-backup-1970-01-01T000000Z/recovery --scope=certs,code,config\n") + expect_command('/opt/puppetlabs/server/bin/pg_dump -Fd -Z3 -j4 -f /user/home/folder/pe-backup-1970-01-01T000000Z/puppetdb/pe-puppetdb.dump.d "sslmode=verify-ca host=postgres user=pe-puppetdb sslcert=/etc/puppetlabs/puppetdb/ssl/primary.cert.pem sslkey=/etc/puppetlabs/puppetdb/ssl/primary.private_key.pem sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem dbname=pe-puppetdb"' + "\n") + expect_command('umask 0077 && cd /user/home/folder && tar -czf /user/home/folder/pe-backup-1970-01-01T000000Z.tar.gz pe-backup-1970-01-01T000000Z && rm -rf /user/home/folder/pe-backup-1970-01-01T000000Z' + "\n") + + expect(run_plan('peadm::backup', { 'targets' => 'primary', 'output_directory' => '/user/home/folder' })).to be_ok + end + it 'runs with backup type custom, classifier only' do expect_task('peadm::backup_classification').with_params({ 'directory' => '/tmp/pe-backup-1970-01-01T000000Z/classifier' }) expect_out_message.with_params('# Backing up classification') diff --git a/spec/plans/restore_spec.rb b/spec/plans/restore_spec.rb index 5df0c5f3..933472be 100644 --- a/spec/plans/restore_spec.rb +++ b/spec/plans/restore_spec.rb @@ -76,6 +76,15 @@ expect(run_plan('peadm::restore', recovery_params)).to be_ok end + it 'runs with default recovery', valid_cluster: true do + allow_any_command + + expect_out_message.with_params('# Restoring database pe-puppetdb') + expect_out_message.with_params('# Restoring ca, certs, code and config for recovery') + + expect(run_plan('peadm::restore', {'targets' => 'primary', 'input_file' => backup_tarball})).to be_ok + end + it 'runs with recovery-db params', valid_cluster: true do allow_any_command From 5c1c4af113929751732e9a576e884791bfd0578b Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko <1586813+timidri@users.noreply.github.com> Date: Mon, 8 Apr 2024 20:54:36 +0200 Subject: [PATCH 092/110] Update documentation/backup_restore.md Co-authored-by: JHunniford <97452751+J-Hunniford@users.noreply.github.com> --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 0e815579..5500f16b 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -134,7 +134,7 @@ The following table shows the items you can specify and indicates what is includ ## Recovering a broken primary server -When your primary server is broken, you might not be able to use the `recovery` restore directly because the primary server is not in an operational state. In this case, follow the steps below to restore your primary server: +If you cannot run the `recovery` restore plan directly because your primary server is not operational, you can use the following process to restore PE: 1. Make sure you have a `recovery` backup of your primary server. 1. Uninstall and reinstall Puppet Enterprise on the broken primary server. Make sure you are installing the same PE version as your current primary server was running. Optionally, you can use the `peadm::reinstall_pe` task as follows: ``` From bffd08622459aa76c4d87b001457c8d0f3e751ad Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko <1586813+timidri@users.noreply.github.com> Date: Mon, 8 Apr 2024 20:56:08 +0200 Subject: [PATCH 093/110] Update documentation/backup_restore.md Co-authored-by: JHunniford <97452751+J-Hunniford@users.noreply.github.com> --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 5500f16b..409598f5 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -136,7 +136,7 @@ The following table shows the items you can specify and indicates what is includ If you cannot run the `recovery` restore plan directly because your primary server is not operational, you can use the following process to restore PE: 1. Make sure you have a `recovery` backup of your primary server. -1. Uninstall and reinstall Puppet Enterprise on the broken primary server. Make sure you are installing the same PE version as your current primary server was running. Optionally, you can use the `peadm::reinstall_pe` task as follows: +1. On the node hosting the broken primary server, uninstall and reinstall PE, ensuring that you re-install the same PE version. Optionally, you can use the `peadm::reinstall_pe` task as follows: ``` bolt task run peadm::reinstall_pe --targets my.primary.vm uninstall=true version=2023.5.0 ``` From 74b6e99af9ab029b904df87320855d46572be9ac Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko <1586813+timidri@users.noreply.github.com> Date: Mon, 8 Apr 2024 20:56:46 +0200 Subject: [PATCH 094/110] Update documentation/backup_restore.md Co-authored-by: JHunniford <97452751+J-Hunniford@users.noreply.github.com> --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 409598f5..0ec171c8 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -140,7 +140,7 @@ If you cannot run the `recovery` restore plan directly because your primary serv ``` bolt task run peadm::reinstall_pe --targets my.primary.vm uninstall=true version=2023.5.0 ``` -1. Perform a `recovery` restore of your primary server using the backup. For example: +1. Perform a `recovery` restore of your primary server, specifying the backup file that you want to use. For example: ``` bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery ``` From e6458f52d5839bf231df7713fe210bb0136b3f67 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko <1586813+timidri@users.noreply.github.com> Date: Mon, 8 Apr 2024 20:57:18 +0200 Subject: [PATCH 095/110] Update documentation/backup_restore.md Co-authored-by: JHunniford <97452751+J-Hunniford@users.noreply.github.com> --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 0ec171c8..fb5c28c8 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -145,7 +145,7 @@ If you cannot run the `recovery` restore plan directly because your primary serv bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery ``` -## Recovering a broken external primary database server in an Extra Large (XL) installation +## Recovering a broken database server in an extra-large installation When your primary database server is broken, you might not be able to use the `recovery` restore directly because the puppetdb database will not be operational. In this case, follow the steps below to restore your primary database: 1. Make sure you have a `recovery` backup of your primary server. From b50d3fe130b251acebec521a7d13db2e003dc936 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko <1586813+timidri@users.noreply.github.com> Date: Mon, 8 Apr 2024 20:57:41 +0200 Subject: [PATCH 096/110] Update documentation/backup_restore.md Co-authored-by: JHunniford <97452751+J-Hunniford@users.noreply.github.com> --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index fb5c28c8..ad5e97af 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -167,7 +167,7 @@ To do this, use the plan `peadm::util::init_db_server` as follows: 1. Start the puppetdb service on the compilers. 1. Restart the puppetserver service on the compilers. -1. Perform a `recovery-db` restore of your primary database server using the backup. For example: +1. Perform a `recovery-db` restore of your database server, specifying the backup file that you want to use. For example: ``` bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery-db ``` From fa0d3eb20e349612f97b771494b7b9af694564a3 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko <1586813+timidri@users.noreply.github.com> Date: Mon, 8 Apr 2024 20:58:20 +0200 Subject: [PATCH 097/110] Update documentation/backup_restore.md Co-authored-by: JHunniford <97452751+J-Hunniford@users.noreply.github.com> --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index ad5e97af..a916c97b 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -171,5 +171,5 @@ To do this, use the plan `peadm::util::init_db_server` as follows: ``` bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery-db ``` - **Note**: You should use the `restore_type=recovery-db` parameter to recover the database server. + **Important**: You must use the `restore_type=recovery-db` parameter to recover the database server. **Note**: You should specify the primary host (not the primary database host) as the target for the restore plan. From 83568f075b45d032ca2380ccc9c2b91d200ae6de Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko <1586813+timidri@users.noreply.github.com> Date: Mon, 8 Apr 2024 20:59:00 +0200 Subject: [PATCH 098/110] Update documentation/backup_restore.md Co-authored-by: JHunniford <97452751+J-Hunniford@users.noreply.github.com> --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index a916c97b..49e96969 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -172,4 +172,4 @@ To do this, use the plan `peadm::util::init_db_server` as follows: bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery-db ``` **Important**: You must use the `restore_type=recovery-db` parameter to recover the database server. - **Note**: You should specify the primary host (not the primary database host) as the target for the restore plan. + **Important**: You must specify the primary server host node (not the database server host node) as the target for the restore plan. From ffd6f286b7bbc5d74261150ba1980f5903f62fd5 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko <1586813+timidri@users.noreply.github.com> Date: Mon, 8 Apr 2024 20:59:13 +0200 Subject: [PATCH 099/110] Update documentation/backup_restore.md Co-authored-by: JHunniford <97452751+J-Hunniford@users.noreply.github.com> --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 49e96969..bfdb21b1 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -6,7 +6,7 @@ - [Using `custom` backup and restore](#using-custom-backup-and-restore) - [What exactly is backed up and restored?](#what-exactly-is-backed-up-and-restored) - [Recovering a broken primary server](#recovering-a-broken-primary-server) - - [Recovering a broken external primary database server in an Extra Large (XL) installation](#recovering-a-broken-external-primary-database-server-in-an-extra-large-xl-installation) + - [Recovering a broken database server in an extra-large installation](#recovering-a-broken-database-server-in-an-extra-large-installation) ## Introduction to PEADM backup and restore From aecb6a52f780dcb3a2896a1dde903d88f322cda2 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko <1586813+timidri@users.noreply.github.com> Date: Mon, 8 Apr 2024 20:59:49 +0200 Subject: [PATCH 100/110] Update documentation/backup_restore.md Co-authored-by: JHunniford <97452751+J-Hunniford@users.noreply.github.com> --- documentation/backup_restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index bfdb21b1..9a7ff7d2 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -22,7 +22,7 @@ When running the backup and restore plans, you can define the `backup_type` and If no type is specified, the default is `recovery`. -(When running the restore plan, an additional restore type `recovery-db` is available to recover a failed primary database server (see [Recovering a broken external primary database server in an Extra Large (XL) installation](#recovering-a-broken-external-primary-database-server-in-an-extra-large-xl-installation)).) +(When running the restore plan, you can use the `recovery-db` restore type to recover a failed database server (see [Recovering a broken database server in an extra-large installation](#recovering-a-broken-database-server-in-an-extra-large-installation)).) When backing up or restoring PE, you must use the `--targets` option to specify the hostname (FQDN) of your primary server. From 334dfe5e954da513ebe6907e007859122dd3e239 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Mon, 8 Apr 2024 21:05:16 +0200 Subject: [PATCH 101/110] Some suggested updates to backup-restore docs. --- documentation/backup_restore.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 9a7ff7d2..ee7306d2 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -134,26 +134,29 @@ The following table shows the items you can specify and indicates what is includ ## Recovering a broken primary server +**Important**: To complete the recovery process outlined here, you must have a recovery backup of your primary server. + If you cannot run the `recovery` restore plan directly because your primary server is not operational, you can use the following process to restore PE: -1. Make sure you have a `recovery` backup of your primary server. 1. On the node hosting the broken primary server, uninstall and reinstall PE, ensuring that you re-install the same PE version. Optionally, you can use the `peadm::reinstall_pe` task as follows: - ``` + ``` bolt task run peadm::reinstall_pe --targets my.primary.vm uninstall=true version=2023.5.0 - ``` + ``` 1. Perform a `recovery` restore of your primary server, specifying the backup file that you want to use. For example: - ``` + ``` bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery - ``` + ``` ## Recovering a broken database server in an extra-large installation +**Important**: To complete the recovery process outlined here, you must have a recovery backup of your primary server. + When your primary database server is broken, you might not be able to use the `recovery` restore directly because the puppetdb database will not be operational. In this case, follow the steps below to restore your primary database: -1. Make sure you have a `recovery` backup of your primary server. + 1. Reinstall Puppet Enterprise on the broken database server and reconfigure and re-sign its certificate. Make sure you are installing the same PE version as your current primary server was running. To do this, use the plan `peadm::util::init_db_server` as follows: - ``` + ``` bolt plan run peadm::util::init_db_server db_host=my.primary_db.vm pe_version=2023.5.0 install_pe=true - ``` + ``` This plan will perform the following: @@ -168,8 +171,8 @@ To do this, use the plan `peadm::util::init_db_server` as follows: 1. Restart the puppetserver service on the compilers. 1. Perform a `recovery-db` restore of your database server, specifying the backup file that you want to use. For example: - ``` + ``` bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery-db - ``` + ``` **Important**: You must use the `restore_type=recovery-db` parameter to recover the database server. **Important**: You must specify the primary server host node (not the database server host node) as the target for the restore plan. From 10f903db574b1886c58ff6f7ea9a209ec30bff14 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Wed, 10 Apr 2024 13:49:09 +0100 Subject: [PATCH 102/110] Updating backup options table --- documentation/backup_restore.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index ee7306d2..048c50be 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -124,9 +124,9 @@ The following table shows the items you can specify and indicates what is includ | `classifier` | Classifier database. Restore merges user-defined node groups rather than overwriting system node groups. | | | `code` | Code directory | ✅ | | `config` | Configuration files and databases (databases are restored literally) | ✅ | -| `orchestrator ` | Orchestrator database | | +| `orchestrator ` | Orchestrator database and secrets | | | `puppetdb` | PuppetDB database (including support for XL where puppetdb is running on an external db server) | ✅ | -| `rbac` | RBAC database | | +| `rbac` | RBAC database and secrets | | **Note**: The PEADM backup and restore plans utilize the `puppet-backup` tool for backing up and restoring `ca`, `code` and `config`. For `config`, the data backed up includes the `activity`, `classifier`, `orchestrator`, and `rbac` databases. From 1c5eede2b4aee6f0c73a2e4e5c0d6288e9590dc6 Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Tue, 16 Apr 2024 16:02:46 +0100 Subject: [PATCH 103/110] update language --- documentation/backup_restore.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 048c50be..df386c0c 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -5,8 +5,8 @@ - [Using `recovery` backup and restore](#using-recovery-backup-and-restore) - [Using `custom` backup and restore](#using-custom-backup-and-restore) - [What exactly is backed up and restored?](#what-exactly-is-backed-up-and-restored) - - [Recovering a broken primary server](#recovering-a-broken-primary-server) - - [Recovering a broken database server in an extra-large installation](#recovering-a-broken-database-server-in-an-extra-large-installation) + - [Recovering a primary server when some or all services are not operational](#recovering-a-primary-server-when-some-or-all-services-are-not-operational) + - [Recovering a non-operational database server in an extra-large installation](#recovering-a-non-operational-database-server-in-an-extra-large-installation) ## Introduction to PEADM backup and restore @@ -124,20 +124,20 @@ The following table shows the items you can specify and indicates what is includ | `classifier` | Classifier database. Restore merges user-defined node groups rather than overwriting system node groups. | | | `code` | Code directory | ✅ | | `config` | Configuration files and databases (databases are restored literally) | ✅ | -| `orchestrator ` | Orchestrator database and secrets | | +| `orchestrator ` | Orchestrator database and secrets | | | `puppetdb` | PuppetDB database (including support for XL where puppetdb is running on an external db server) | ✅ | -| `rbac` | RBAC database and secrets | | +| `rbac` | RBAC database and secrets | | **Note**: The PEADM backup and restore plans utilize the `puppet-backup` tool for backing up and restoring `ca`, `code` and `config`. For `config`, the data backed up includes the `activity`, `classifier`, `orchestrator`, and `rbac` databases. **Note:** The output for the `peadm::backup` plan differs from the output that is returned when you manually run the [`puppet-backup create` command](https://puppet.com/docs/pe/latest/backing_up_and_restoring_pe.html#back_up_pe_infrastructure). -## Recovering a broken primary server +## Recovering a primary server when some or all services are not operational **Important**: To complete the recovery process outlined here, you must have a recovery backup of your primary server. If you cannot run the `recovery` restore plan directly because your primary server is not operational, you can use the following process to restore PE: -1. On the node hosting the broken primary server, uninstall and reinstall PE, ensuring that you re-install the same PE version. Optionally, you can use the `peadm::reinstall_pe` task as follows: +1. On the node hosting the affected primary server, uninstall and reinstall PE, ensuring that you re-install the same PE version. Optionally, you can use the `peadm::reinstall_pe` task as follows: ``` bolt task run peadm::reinstall_pe --targets my.primary.vm uninstall=true version=2023.5.0 ``` @@ -146,13 +146,13 @@ If you cannot run the `recovery` restore plan directly because your primary serv bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery ``` -## Recovering a broken database server in an extra-large installation +## Recovering a non-operational database server in an extra-large installation **Important**: To complete the recovery process outlined here, you must have a recovery backup of your primary server. -When your primary database server is broken, you might not be able to use the `recovery` restore directly because the puppetdb database will not be operational. In this case, follow the steps below to restore your primary database: +When your primary database server is not operational, you might not be able to use the `recovery` restore directly because the puppetdb database service will not be operational. In this case, follow the steps below to restore your primary database: -1. Reinstall Puppet Enterprise on the broken database server and reconfigure and re-sign its certificate. Make sure you are installing the same PE version as your current primary server was running. +1. Reinstall Puppet Enterprise on the affected database server and reconfigure and re-sign its certificate. Make sure you are installing the same PE version as your current primary server was running. To do this, use the plan `peadm::util::init_db_server` as follows: ``` bolt plan run peadm::util::init_db_server db_host=my.primary_db.vm pe_version=2023.5.0 install_pe=true From da3e07cd783a9245987027adc46f241d0d4fda8a Mon Sep 17 00:00:00 2001 From: Dimitri Tischenko Date: Wed, 15 May 2024 12:24:24 +0200 Subject: [PATCH 104/110] add info on hostname equality --- documentation/backup_restore.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index df386c0c..1e5f821a 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -25,6 +25,8 @@ If no type is specified, the default is `recovery`. (When running the restore plan, you can use the `recovery-db` restore type to recover a failed database server (see [Recovering a broken database server in an extra-large installation](#recovering-a-broken-database-server-in-an-extra-large-installation)).) When backing up or restoring PE, you must use the `--targets` option to specify the hostname (FQDN) of your primary server. +**Note**: the hostname of your primary server you are restoring to _must be equal_ to the hostname of the primary server you created the backup from. +In other words, you can't change the hostname of your primary server during the recovery process. The backup file created is named `pe-backup-YYYY-MM-DDTHHMMSSZ.tar.gz` and placed by default in `/tmp`. To specify a different location for the backup file, you can define the `output_directory` parameter. From e29233a1b24e02f0e498e4d37e26ae5d714ad56a Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Tue, 21 May 2024 08:32:49 +0100 Subject: [PATCH 105/110] Updating output message to remote quote, updating docs to include pe_platform and removing comma in custom restore --- documentation/backup_restore.md | 4 ++-- plans/restore.pp | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 1e5f821a..40234977 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -104,7 +104,7 @@ To specify custom restore options: "config" : true, "orchestrator" : false, "puppetdb" : true, - "rbac" : false, + "rbac" : false }, "input_file" : "/tmp/my_backup.tar.gz" } @@ -157,7 +157,7 @@ When your primary database server is not operational, you might not be able to u 1. Reinstall Puppet Enterprise on the affected database server and reconfigure and re-sign its certificate. Make sure you are installing the same PE version as your current primary server was running. To do this, use the plan `peadm::util::init_db_server` as follows: ``` - bolt plan run peadm::util::init_db_server db_host=my.primary_db.vm pe_version=2023.5.0 install_pe=true + bolt plan run peadm::util::init_db_server db_host=my.primary_db.vm pe_version=2023.5.0 install_pe=true pe_platform=el-8-x86_64 ``` This plan will perform the following: diff --git a/plans/restore.pp b/plans/restore.pp index d0f5811a..453bf8e5 100644 --- a/plans/restore.pp +++ b/plans/restore.pp @@ -196,7 +196,7 @@ run_command(@("CMD"/L), $primary_target) test -f ${shellquote($recovery_directory)}/rbac/keys.json \ && cp -rp ${shellquote($recovery_directory)}/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ \ - || echo secret ldap key doesn't exist + || echo secret ldap key doesnt exist | CMD # lint:ignore:140chars # IF restoring orchestrator restore the secrets to /etc/puppetlabs/orchestration-services/conf.d/secrets/ From bffe3f8f6d1ad46db34dc355b5f75f0d91fe8d4e Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Thu, 30 May 2024 17:22:16 +0100 Subject: [PATCH 106/110] Fixing rubocop issues Regenerated reference doc Added spaces to params in spec file Changed commit reference to version for node_manager --- Puppetfile | 4 +- REFERENCE.md | 209 ++++++++++++++++++++++++++++++++++++- spec/plans/restore_spec.rb | 2 +- 3 files changed, 206 insertions(+), 9 deletions(-) diff --git a/Puppetfile b/Puppetfile index d304ab39..9cbccda3 100644 --- a/Puppetfile +++ b/Puppetfile @@ -4,8 +4,6 @@ # The following directive installs modules to the managed moduledir. moduledir '.modules' -mod 'node_manager', - git: 'https://github.com/WhatsARanjit/puppet-node_manager', - ref: '86cb48f27a4a0ea212b33f86775d84a374a189b7' +mod 'WhatsARanjit/node_manager', '0.7.5' mod 'puppet/format', '1.1.1' mod 'puppetlabs/stdlib', '9.5.0' diff --git a/REFERENCE.md b/REFERENCE.md index ee1a02a0..ac9180c5 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -31,9 +31,11 @@ * [`peadm::generate_pe_conf`](#peadm--generate_pe_conf): Generate a pe.conf file in JSON format * [`peadm::get_pe_conf`](#peadm--get_pe_conf) * [`peadm::get_targets`](#peadm--get_targets): Accept undef or a SingleTargetSpec, and return an Array[Target, 1, 0]. This differs from get_target() in that: - It returns an Array[Target +* [`peadm::migration_opts_default`](#peadm--migration_opts_default) * [`peadm::node_manager_yaml_location`](#peadm--node_manager_yaml_location) * [`peadm::oid`](#peadm--oid) * [`peadm::plan_step`](#peadm--plan_step) +* [`peadm::recovery_opts_all`](#peadm--recovery_opts_all) * [`peadm::recovery_opts_default`](#peadm--recovery_opts_default) * [`peadm::update_pe_conf`](#peadm--update_pe_conf): Update the pe.conf file on a target with the provided hash * [`peadm::wait_until_service_ready`](#peadm--wait_until_service_ready): A convenience function to help remember port numbers for services and handle running the wait_until_service_ready task @@ -73,6 +75,7 @@ * [`puppet_runonce`](#puppet_runonce): Run the Puppet agent one time * [`rbac_token`](#rbac_token): Get and save an rbac token for the root user, admin rbac user * [`read_file`](#read_file): Read the contents of a file +* [`reinstall_pe`](#reinstall_pe): Reinstall PE, only to be used to restore PE * [`restore_classification`](#restore_classification): A short description of this task * [`sign_csr`](#sign_csr): Submit a certificate signing request * [`ssl_clean`](#ssl_clean): Clean an agent's certificate @@ -85,13 +88,16 @@ #### Public Plans * [`peadm::add_database`](#peadm--add_database) +* [`peadm::backup`](#peadm--backup): Backup puppet primary configuration * [`peadm::backup_ca`](#peadm--backup_ca) * [`peadm::convert`](#peadm--convert): Convert an existing PE cluster to a PEAdm-managed cluster * [`peadm::install`](#peadm--install): Install a new PE cluster * [`peadm::modify_certificate`](#peadm--modify_certificate): Modify the certificate of one or more targets +* [`peadm::restore`](#peadm--restore): Restore puppet primary configuration * [`peadm::restore_ca`](#peadm--restore_ca) * [`peadm::status`](#peadm--status): Return status information from one or more PE clusters in a table format * [`peadm::upgrade`](#peadm--upgrade): Upgrade a PEAdm-managed cluster +* [`peadm::util::init_db_server`](#peadm--util--init_db_server) #### Private Plans @@ -99,10 +105,8 @@ * `peadm::add_replica`: Replace a replica host for a Standard or Large architecture. Supported use cases: 1: The existing replica is broken, we have a fresh new VM we want to provision the replica to. -* `peadm::backup`: Backup the core user settings for puppet infrastructure * `peadm::misc::divert_code_manager`: This plan exists to account for a scenario where a PE XL * `peadm::modify_cert_extensions` -* `peadm::restore`: Restore the core user settings for puppet infrastructure from backup * `peadm::subplans::component_install`: Install a new PEADM component * `peadm::subplans::configure`: Configure first-time classification and DR setup * `peadm::subplans::db_populate`: Destructively (re)populates a new or existing database with the contents or a known good source @@ -758,6 +762,18 @@ Data type: `Optional[Integer[1,1]]` +### `peadm::migration_opts_default` + +Type: Puppet Language + +The peadm::migration_opts_default function. + +#### `peadm::migration_opts_default()` + +The peadm::migration_opts_default function. + +Returns: `Any` + ### `peadm::node_manager_yaml_location` Type: Ruby 4.x API @@ -812,6 +828,18 @@ Data type: `Callable` +### `peadm::recovery_opts_all` + +Type: Puppet Language + +The peadm::recovery_opts_all function. + +#### `peadm::recovery_opts_all()` + +The peadm::recovery_opts_all function. + +Returns: `Any` + ### `peadm::recovery_opts_default` Type: Puppet Language @@ -947,12 +975,14 @@ Alias of ```puppet Struct[{ - 'orchestrator' => Optional[Boolean], - 'puppetdb' => Optional[Boolean], - 'rbac' => Optional[Boolean], 'activity' => Optional[Boolean], 'ca' => Optional[Boolean], 'classifier' => Optional[Boolean], + 'code' => Optional[Boolean], + 'config' => Optional[Boolean], + 'orchestrator' => Optional[Boolean], + 'puppetdb' => Optional[Boolean], + 'rbac' => Optional[Boolean], }] ``` @@ -1394,6 +1424,32 @@ Data type: `String` Path to the file to read +### `reinstall_pe` + +Reinstall PE, only to be used to restore PE + +**Supports noop?** false + +#### Parameters + +##### `version` + +Data type: `String[1]` + +The PE version to install + +##### `arch` + +Data type: `String[1]` + +The PE installation platform + +##### `uninstall` + +Data type: `Boolean` + +Whether we want to uninstall PE before installing + ### `restore_classification` A short description of this task @@ -1543,6 +1599,57 @@ Optional[Enum[ Default value: `undef` +### `peadm::backup` + +Backup puppet primary configuration + +#### Examples + +##### + +```puppet +bolt plan run peadm::backup -t primary1.example.com +``` + +#### Parameters + +The following parameters are available in the `peadm::backup` plan: + +* [`targets`](#-peadm--backup--targets) +* [`backup_type`](#-peadm--backup--backup_type) +* [`backup`](#-peadm--backup--backup) +* [`output_directory`](#-peadm--backup--output_directory) + +##### `targets` + +Data type: `Peadm::SingleTargetSpec` + +This should be the primary puppetserver for the puppet cluster + +##### `backup_type` + +Data type: `Enum['recovery', 'custom']` + +Currently, the recovery and custom backup types are supported + +Default value: `'recovery'` + +##### `backup` + +Data type: `Peadm::Recovery_opts` + +A hash of custom backup options, see the peadm::recovery_opts_default() function for the default values + +Default value: `{}` + +##### `output_directory` + +Data type: `String` + +The directory to place the backup in + +Default value: `'/tmp'` + ### `peadm::backup_ca` The peadm::backup_ca class. @@ -2006,6 +2113,55 @@ Data type: `Boolean` Default value: `false` +### `peadm::restore` + +Restore puppet primary configuration + +#### Examples + +##### + +```puppet +bolt plan run peadm::restore -t primary1.example.com input_file=/tmp/peadm-backup.tar.gz +``` + +#### Parameters + +The following parameters are available in the `peadm::restore` plan: + +* [`targets`](#-peadm--restore--targets) +* [`restore_type`](#-peadm--restore--restore_type) +* [`restore`](#-peadm--restore--restore) +* [`input_file`](#-peadm--restore--input_file) + +##### `targets` + +Data type: `Peadm::SingleTargetSpec` + +This should be the primary puppetserver for the puppet cluster + +##### `restore_type` + +Data type: `Enum['recovery', 'recovery-db', 'custom']` + +Choose from `recovery`, `recovery-db` and `custom` + +Default value: `'recovery'` + +##### `restore` + +Data type: `Peadm::Recovery_opts` + +A hash of custom backup options, see the peadm::recovery_opts_default() function for the default values + +Default value: `{}` + +##### `input_file` + +Data type: `Pattern[/.*\.tar\.gz$/]` + +The file containing the backup to restore from + ### `peadm::restore_ca` The peadm::restore_ca class. @@ -2291,3 +2447,46 @@ Optional[Enum[ Default value: `undef` +### `peadm::util::init_db_server` + +The peadm::util::init_db_server class. + +#### Parameters + +The following parameters are available in the `peadm::util::init_db_server` plan: + +* [`db_host`](#-peadm--util--init_db_server--db_host) +* [`install_pe`](#-peadm--util--init_db_server--install_pe) +* [`pe_version`](#-peadm--util--init_db_server--pe_version) +* [`pe_platform`](#-peadm--util--init_db_server--pe_platform) + +##### `db_host` + +Data type: `String[1]` + + + +##### `install_pe` + +Data type: `Boolean` + + + +Default value: `false` + +##### `pe_version` + +Data type: `String[1]` + + + +Default value: `'2023.5.0'` + +##### `pe_platform` + +Data type: `String[1]` + + + +Default value: `'el-8-x86_64'` + diff --git a/spec/plans/restore_spec.rb b/spec/plans/restore_spec.rb index 933472be..01138dc1 100644 --- a/spec/plans/restore_spec.rb +++ b/spec/plans/restore_spec.rb @@ -82,7 +82,7 @@ expect_out_message.with_params('# Restoring database pe-puppetdb') expect_out_message.with_params('# Restoring ca, certs, code and config for recovery') - expect(run_plan('peadm::restore', {'targets' => 'primary', 'input_file' => backup_tarball})).to be_ok + expect(run_plan('peadm::restore', { 'targets' => 'primary', 'input_file' => backup_tarball })).to be_ok end it 'runs with recovery-db params', valid_cluster: true do From 697c1b4b540626c40ca1626ae39c8ef675679207 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Thu, 30 May 2024 17:40:38 +0100 Subject: [PATCH 107/110] Fixing expected text in spec test --- spec/plans/restore_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/plans/restore_spec.rb b/spec/plans/restore_spec.rb index 01138dc1..52754b8e 100644 --- a/spec/plans/restore_spec.rb +++ b/spec/plans/restore_spec.rb @@ -64,7 +64,7 @@ expect_command("umask 0077 && cd /input && tar -xzf /input/file.tar.gz\n") expect_command("/opt/puppetlabs/bin/puppet-backup restore --scope=certs,code,config --tempdir=/input/file --force /input/file/recovery/pe_backup-*tgz\n") expect_command("systemctl stop pe-console-services pe-nginx pxp-agent pe-puppetserver pe-orchestration-services puppet pe-puppetdb\n") - expect_command("test -f /input/file/rbac/keys.json && cp -rp /input/file/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ || echo secret ldap key doesn't exist\n") + expect_command("test -f /input/file/rbac/keys.json && cp -rp /input/file/keys.json /etc/puppetlabs/console-services/conf.d/secrets/ || echo secret ldap key doesnt exist\n") expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql --tuples-only -d 'pe-puppetdb' -c 'DROP SCHEMA IF EXISTS pglogical CASCADE;'\"\n").be_called_times(2) expect_command("su - pe-postgres -s /bin/bash -c \"/opt/puppetlabs/server/bin/psql -d 'pe-puppetdb' -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'\"\n") expect_command('su - pe-postgres -s /bin/bash -c "/opt/puppetlabs/server/bin/psql -d \'pe-puppetdb\' -c \'ALTER USER \\"pe-puppetdb\\" WITH SUPERUSER;\'"' + "\n") From e71d9e16a4fdd62b5771f5b20220eab25c0146b0 Mon Sep 17 00:00:00 2001 From: Neil Anderson Date: Mon, 3 Jun 2024 11:43:59 +0100 Subject: [PATCH 108/110] Removing todo comments --- .github/workflows/test-backup-restore.yaml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index 1390663b..a67812f2 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -1,8 +1,4 @@ --- -# TODO: -# - [ ] test on XL, also break the puppetdb database -# - [ ] test custom scenario for only restoring primary -# - [ ] test custom scenario for only restoring puppetdb (on external server) name: "Backup and restore test" on: @@ -191,8 +187,6 @@ jobs: --modulepath spec/fixtures/modules \ --stream - # TODO: add docs to README about actions the customer needs to do to recover the primary db - - name: Break external DB if: ${{ github.event.inputs.architecture || 'extra-large' }} == 'extra-large' run: | @@ -244,7 +238,6 @@ jobs: - name: "Wait as long as the file ${HOME}/pause file is present" continue-on-error: true if: ${{ always() && github.event.inputs.ssh-debugging == 'true' }} - # if: github.event.inputs.ssh-debugging == 'true' run: | while [ -f "${HOME}/pause" ] ; do echo "${HOME}/pause present, sleeping for 60 seconds..." From 376c65e3c789409fd99fe7ca99f84b910fb2f77e Mon Sep 17 00:00:00 2001 From: Ben Ford Date: Tue, 25 Jun 2024 10:03:29 -0700 Subject: [PATCH 109/110] [ITHELP-87329] Update test-backup-restore.yaml (#447) This looks like it should work with the `pull_request` event --- .github/workflows/test-backup-restore.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test-backup-restore.yaml b/.github/workflows/test-backup-restore.yaml index a67812f2..e7fce6de 100644 --- a/.github/workflows/test-backup-restore.yaml +++ b/.github/workflows/test-backup-restore.yaml @@ -2,7 +2,7 @@ name: "Backup and restore test" on: - pull_request_target: + pull_request: types: [ready_for_review] workflow_dispatch: From 9c1a20d30811015650cedb12ca65b231cae4ffaa Mon Sep 17 00:00:00 2001 From: JHunniford <97452751+J-Hunniford@users.noreply.github.com> Date: Wed, 26 Jun 2024 08:31:31 +0100 Subject: [PATCH 110/110] Update backup_restore.md (#432) * Update backup_restore.md * fix typo * Update documentation/backup_restore.md * Update backup_restore.md Integrating the extra note (added to SOLARCH 1160 branch) that was causing the conflict into my changes. * Update backup_restore.md * [ITHELP-87329] Update test-backup-restore.yaml (#446) This looks like it should work with the `pull_request` event --------- Co-authored-by: Ben Ford --- documentation/backup_restore.md | 92 +++++++++++++-------------------- 1 file changed, 37 insertions(+), 55 deletions(-) diff --git a/documentation/backup_restore.md b/documentation/backup_restore.md index 40234977..4ad28012 100644 --- a/documentation/backup_restore.md +++ b/documentation/backup_restore.md @@ -14,63 +14,46 @@ If your PE installation is managed by PEADM, you can back up and restore PE usin 1. Use the `peadm::backup` plan to create a backup of your primary server. 2. Use the `peadm::restore` plan to restore PE from a `peadm::backup`. -**Important:** If your PE installation is not managed by PEADM, you cannot use the `peadm::backup` and `peadm::restore` plans. For information on converting to a PEADM-managed installation, see [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md). +**Important:** If your PE installation is not managed by PEADM, you cannot use the `peadm::backup` and `peadm::restore` plans. For information about converting to a PEADM-managed installation, see [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md). -When running the backup and restore plans, you can define the `backup_type` and `restore_type` parameters with either of the following values: -* `recovery`: Use this type to create a full backup of your primary server, including data for all services. This allows you to restore your primary server and all services (including database services running on external servers) to the exact state they were in at the time of the backup. -* `custom`: Use this type when you want to selectively back up and restore data for specific services. +You can specify the type of backup or restore plan you want to use. There are two types: +- `recovery`: Use this type to create a full backup of your primary server, including data for all services. The recovery option allows you to restore your primary server and all services (including database services running on external servers) to the exact state they were in at the time of the backup. +- `custom`: Use this type when you want to back up and restore data for specific services. If no type is specified, the default is `recovery`. -(When running the restore plan, you can use the `recovery-db` restore type to recover a failed database server (see [Recovering a broken database server in an extra-large installation](#recovering-a-broken-database-server-in-an-extra-large-installation)).) - -When backing up or restoring PE, you must use the `--targets` option to specify the hostname (FQDN) of your primary server. -**Note**: the hostname of your primary server you are restoring to _must be equal_ to the hostname of the primary server you created the backup from. -In other words, you can't change the hostname of your primary server during the recovery process. - -The backup file created is named `pe-backup-YYYY-MM-DDTHHMMSSZ.tar.gz` and placed by default in `/tmp`. To specify a different location for the backup file, you can define the `output_directory` parameter. - -This example shows how to run a `recovery` backup which places the backup file in a custom location. -``` -bolt plan run peadm::backup --targets my.primary.vm backup_type=recovery output_directory=/custom_path -``` - -When restoring PE, you must define the `input_file` parameter to specify the path to the backup file you want to use. For example: -``` -bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" -``` +**Important**: When restoring your installation, the hostname of the primary server you are restoring to _must be the same as_ the hostname of the primary server you created the backup from. +You cannot successfully restore your installation if you change the hostname of your primary server during the recovery process. ## Using `recovery` backup and restore When you run a `recovery` backup plan, the primary server configuration is backed up in full. In the event of a primary server failure, this backup can be used to to restore your primary server and all services (including database services running on external servers) to the exact state they were in at the time of the backup. -You can create a `recovery` backup as follows: +### Create a `recovery` backup +Run one of the following commands: +- To create the backup file in the default location, run the `peadm::backup` plan, including the `--targets` option to specify the hostname (FQDN) of your primary server: ``` bolt plan run peadm::backup --targets my.primary.vm backup_type=recovery ``` -Alternatively, because `recovery` is the default type, you can use this simplified command: +- Alternatively, because `recovery` is the default type, you can use this simplified command: ``` bolt plan run peadm::backup --targets my.primary.vm ``` - -To restore your installation from this backup, run: +- To place the backup file in a custom location, define the `output_directory` parameter. For example: +``` +bolt plan run peadm::backup --targets my.primary.vm backup_type=recovery output_directory=/custom_path +``` +### Restore your installation from a `recovery` backup +Run the `peadm::restore` plan, including the `--targets` option to specify the hostname (FQDN) of your primary server and defining the `input_file` parameter to specify the path to the backup file you want to use. For example. ``` bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" ``` - -**Tip**: Restoring from a `recovery` backup restarts any services that are unavailable on the primary server. +**Note**: Restoring from a `recovery` backup restarts any services that are unavailable on the primary server. ## Using `custom` backup and restore - -To specify the items that are backed up and restored, define the `backup_type` or `restore_type` parameters as `custom`. -Otherwise, the default type is `recovery`. - -**Note:** To customize the list of items that are backed up and restored, you must define the `backup` and `restore` parameters, specifying the items you want to exclude. - -To specify the `custom` items, you can create and reference `params.json` files as shown in the following examples. - -To specify custom backup options: -```json +### Create a `custom` backup. +To customize the items you back up, first create a JSON file in which you define the `backup_type` parameter as `custom` and define the `backup` parameter by specifying which items you want to exclude. For example: +``` { "backup_type" : "custom", "backup": { @@ -85,15 +68,13 @@ To specify custom backup options: } } ``` - -To create a backup using the options specified in this parameter file, run: +When you have created the JSON file specifying your custom backup, run the `peadm::backup` plan, including the `--params` option and specifying the relevant filename (and file path if necessary). For example: ``` bolt plan run peadm::backup --targets my.primary.vm --params @params.json ``` - -To specify custom restore options: - -```json +### Restore custom items +To customize the items you restore, create a JSON file in which you define the `restore_type` parameter as `custom`, define the `restore` parameter by specifying the items you want to exclude, and define the `input_file` parameter by specifying the path to the relevant backup file. For example, +``` { "restore_type" : "custom", "restore": { @@ -110,7 +91,7 @@ To specify custom restore options: } ``` -To restore PE using the options specified in this parameter file, run: +When you have created the JSON file specifying your custom restore options, run the `peadm::restore` plan, including the `--params` option and specifying the relevant filename (and file path if necessary). For example: ``` bolt plan run peadm::restore --targets my.primary.vm --params @params.json ``` @@ -160,21 +141,22 @@ To do this, use the plan `peadm::util::init_db_server` as follows: bolt plan run peadm::util::init_db_server db_host=my.primary_db.vm pe_version=2023.5.0 install_pe=true pe_platform=el-8-x86_64 ``` - This plan will perform the following: + This plan performs the following tasks: - 1. Clean the current certificate for the database server from the primary server. - 1. Request a new certificate for the database server with the right extensions (peadm_role = puppet/puppetdb-database, peadm_availability_group=A). - 1. Stop the puppetdb service on the compilers. - 1. Prepare a `pe.conf` file on the database server for database installation - 1. Install PE on the database server using the generated `pe.conf` file. - 1. Configure the database as the primary puppetdb database in the XL installation. - 1. Run puppet on the compilers to allow puppetdb on the compilers to be reconfigured with the new primary database server. - 1. Start the puppetdb service on the compilers. - 1. Restart the puppetserver service on the compilers. + 1. Cleans the current certificate for the database server from the primary server. + 1. Requests a new certificate for the database server with the right extensions (`peadm_role = puppet/puppetdb-database`, `peadm_availability_group=A`). + 1. Stops the puppetdb service on the compilers. + 1. Prepares a `pe.conf` file on the database server for database installation + 1. Installs PE on the database server using the generated `pe.conf` file. + 1. Configures the database as the primary puppetdb database in the XL installation. + 1. Runs puppet on the compilers to allow puppetdb on the compilers to be reconfigured with the new primary database server. + 1. Starts the puppetdb service on the compilers. + 1. Restarts the puppetserver service on the compilers. 1. Perform a `recovery-db` restore of your database server, specifying the backup file that you want to use. For example: ``` bolt plan run peadm::restore --targets my.primary.vm input_file="/tmp/my_backup.tar.gz" restore_type=recovery-db ``` - **Important**: You must use the `restore_type=recovery-db` parameter to recover the database server. + **Important**: You must use the `restore_type=recovery-db` parameter to recover the database server. + **Important**: You must specify the primary server host node (not the database server host node) as the target for the restore plan.