Add backup/restore plans #339

timidri · 2023-02-17T15:36:17Z

Summary

Adding backup/restore functionality

Checklist

🟢 Spec tests.
🟢 Acceptance tests.

Changes include test coverage?

Yes
Not needed

Have you updated the documentation?

Yes, I've updated the appropriate docs
Not needed

plans/restore.pp

davidsandilands · 2023-03-21T10:50:03Z

https://tickets.puppetlabs.com/browse/PE-35454?jql=text%20~%20%22puppet%20backup%20cert%22%20ORDER%20BY%20created%20DESC is now resolved so this could be tested when we have resource

cdenneen · 2023-06-29T13:28:52Z

Would like to leverage backup/restore for few pieces:

Have existing 2019.8.12 stack (ie. pe-2019.8.12-(primary,replica).us-east-1.domain.com)
Backup and restore, increment serial, upgrade pe-2021.7.4-(primary,replica).us-east-1.domain.com (Assuming the host/certnames would need to change in the Node Classification.

Trying to work out the steps if that can be done as:

new 2021.7.4 infra with a CA copy, DB copy.
build new stack (2021.7.4) but as (2019.8.12), backup/restore, increment serial, perform upgrade

After flipping stacks from old -> new we can do another task to rsync the remaining client certs from old CA to new (with incremented serial this should avoid any overlap (Thanks @Sharpie)).

bastelfreak · 2023-11-21T12:26:30Z

.github/workflows/test-backup-restore.yaml

@@ -5,42 +5,33 @@ on:
  workflow_dispatch:
    inputs:
      image:
-        description: 'GCP image for test cluster'


why is all the quoting changed? I think the change isn't related to adding the backup/restore plan. If it's required it should be done in another PR, to reduce the diff here.

@bastelfreak The integration testing is actually part of this PR since that is needed to make backup/restore releasable.
The quoting change is an unfortunate side-effect of auto-formatting I'm afraid... We will work on synchronizing auto-formatting rules since this is annoying, thanks for catching this.

now this was merged with the quote changes. Now the style in the repo is very inconsistent and the diff in this PR is unnecessary big.

Inconsistency was inevitable from this change, and will be synchronised in a separate PR.

…platform and removing comma in custom restore

Regenerated reference doc Added spaces to params in spec file Changed commit reference to version for node_manager

This looks like it should work with the `pull_request` event

* Update backup_restore.md * fix typo * Update documentation/backup_restore.md * Update backup_restore.md Integrating the extra note (added to SOLARCH 1160 branch) that was causing the conflict into my changes. * Update backup_restore.md * [ITHELP-87329] Update test-backup-restore.yaml (#446) This looks like it should work with the `pull_request` event --------- Co-authored-by: Ben Ford <[email protected]>

bastelfreak · 2024-07-01T16:44:23Z

.github/workflows/test-backup-restore-migration.yaml

+          NGROK_AUTH_TOKEN: ${{ secrets.NGROK_AUTH_TOKEN }}
+          SSH_PASS: ${{ secrets.SSH_PASS }}
+
+          # - name: Download artifacts


Why was that committed if it isn't used?

This could have been moved to another branch until migration was picked up, but wasn't doing any harm for the time being.

bastelfreak · 2024-07-01T16:48:10Z

Gemfile

@@ -35,7 +35,7 @@ group :development do
  gem "rubocop-performance", '= 1.16.0',         require: false
  gem "rubocop-rspec", '= 2.19.0',               require: false
  gem "rb-readline", '= 0.5.5',                  require: false, platforms: [:mswin, :mingw, :x64_mingw]
-  gem "bolt", '>= 3.10.0',                       require: false
+  gem "bolt", '>= 3.27.2',                       require: false


Why is bolt updated? Do old versions not work anymore with PEADM? That's not documented in the backup_restore.md nor the README.md

We felt that it was good to increment the version to keep it within the last couple of years. Not that is wont work anymore, but just difficult for us to sign off support for every version.

bastelfreak · 2024-07-01T16:48:28Z

Puppetfile

@@ -0,0 +1,9 @@
+# This Puppetfile is managed by Bolt. Do not edit.


Why was The Puppetfile committed?

As I'd mentioned elsewhere, this was for the intention of improving instructions for installation and usage.

bastelfreak · 2024-07-01T16:48:53Z

bolt-project.yaml

@@ -0,0 +1,9 @@
+---


why was this file added?

Similar to the Puppetfile

bastelfreak · 2024-07-01T16:49:31Z

functions/migration_opts_default.pp

@@ -0,0 +1,12 @@
+function peadm::migration_opts_default () {


none of the new functions are documented :(

I think these are relatively simple functions for just holding an object, and didn't feel they needed documented.

bastelfreak · 2024-07-01T16:50:37Z

plans/backup.pp

+  Peadm::Recovery_opts        $backup = {},
+
+  # Where to put the backup folder
+  String                      $output_directory = '/tmp',


are relative paths okay, or should this be Stdlib::Absolutepath? WHen relative paths are okay, this should at least be String[1].

I think this is a good idea, and I'll change to this

bastelfreak · 2024-07-01T16:56:21Z

plans/restore.pp

+
+  ## shutdown services
+  run_command(@("CMD"/L), $primary_target)
+    systemctl stop pe-console-services pe-nginx pxp-agent pe-puppetserver \


Is this supposed to stop all services? Because PE provides more. The dead Jira also had a feature request for PE to provide a pe.target that would allow us to reliable stop all services.

This is stopping all the services that we require for the time being.

bastelfreak · 2024-07-01T16:59:37Z

plans/util/init_db_server.pp

@@ -0,0 +1,91 @@
+plan peadm::util::init_db_server(


puppet-strings documentation is missing :(

I think this should maybe be private. I will update documentation for this.

bastelfreak · 2024-07-01T17:02:01Z

tasks/reinstall_pe.sh

+
+# Install PE. We need to pass "y" through stdin since the flag -y requires pe.conf to be present.
+cd $INSTALLER
+echo 'y' | ./puppet-enterprise-installer


trailing newline is missing

golflimaechoecho reviewed Feb 28, 2023

View reviewed changes

plans/restore.pp Outdated Show resolved Hide resolved

timidri added 2 commits April 12, 2023 15:10

Re-add backup/restore plans

564cafe

first WIP version of test plan and workflow

9e29589

timidri force-pushed the SOLARCH-1160 branch from dd0e83d to 9e29589 Compare April 12, 2023 13:11

timidri added 17 commits April 12, 2023 15:15

remove buildevents refs

05c5009

copy inventory file for teardown

42ee72f

fix typo

af03f37

don't start ssh session during provisioning

d134538

cleanup even if cancelled

b39cec0

wait for pause even if previous step failed

a0414da

try to wait before tearing down

791a2db

add continue on error

ca063d6

another attempt

34311d4

removed always()

2db76d5

don't wait in the ssh loop if previous job was cancelled

eb86c03

continue on error

293c5a4

now looking at correct pause file

fc5d215

refactor into 2 jobs

d3bcb83

fix check wait; fix indentation

1a9deb8

fix conditions

8d50129

backup test WIP

7214920

timidri added 5 commits November 8, 2023 15:15

Merge main

8292db3

allow for ca backup

01b23db

change implementation to ruby

0d4d485

formatting

9c764db

typo and path fixes, use cert auth for pdb export

15ad2df

bastelfreak reviewed Nov 21, 2023

View reviewed changes

update REFERENCE.md

b12008a

ragingra and others added 7 commits April 10, 2024 13:49

Updating backup options table

10f903d

update language

1c5eede

add info on hostname equality

da3e07c

Updating output message to remote quote, updating docs to include pe_…

e29233a

…platform and removing comma in custom restore

Merge branch 'main' into SOLARCH-1160

d17a2db

Fixing rubocop issues

bffe3f8

Regenerated reference doc Added spaces to params in spec file Changed commit reference to version for node_manager

Fixing expected text in spec test

697c1b4

ragingra marked this pull request as ready for review May 31, 2024 10:04

ragingra requested review from a team as code owners May 31, 2024 10:04

ragingra and others added 4 commits June 3, 2024 11:43

Removing todo comments

e71d9e1

[ITHELP-87329] Update test-backup-restore.yaml (#447)

376c65e

This looks like it should work with the `pull_request` event

Merge branch 'main' into SOLARCH-1160

7606aa5

ragingra approved these changes Jun 27, 2024

View reviewed changes

ragingra merged commit 9cb918e into main Jun 28, 2024
48 checks passed

ragingra deleted the SOLARCH-1160 branch June 28, 2024 08:45

ragingra added the feature label Jul 1, 2024

bastelfreak reviewed Jul 1, 2024

View reviewed changes

This was referenced Jul 2, 2024

Adding support for legacy compilers #448

Merged

PE-38219 - Support air gapped installation while using a Windows as Jump host #438

Merged

bastelfreak mentioned this pull request Jul 11, 2024

(PE-38155) Updating node_manager to use Latest Puppetlabs version #451

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add backup/restore plans #339

Add backup/restore plans #339

timidri commented Feb 17, 2023 •

edited by ragingra

Loading

davidsandilands commented Mar 21, 2023

cdenneen commented Jun 29, 2023

bastelfreak Nov 21, 2023

timidri Nov 21, 2023 •

edited

Loading

bastelfreak Jul 1, 2024

ragingra Jul 11, 2024

bastelfreak Jul 1, 2024

ragingra Jul 11, 2024

bastelfreak Jul 1, 2024

ragingra Jul 11, 2024

bastelfreak Jul 1, 2024

ragingra Jul 11, 2024

bastelfreak Jul 1, 2024

ragingra Jul 11, 2024

bastelfreak Jul 1, 2024

ragingra Jul 11, 2024

bastelfreak Jul 1, 2024 •

edited

Loading

ragingra Jul 11, 2024

bastelfreak Jul 1, 2024 •

edited

Loading

ragingra Jul 11, 2024

bastelfreak Jul 1, 2024

ragingra Jul 11, 2024

bastelfreak Jul 1, 2024

		@@ -0,0 +1,9 @@
		# This Puppetfile is managed by Bolt. Do not edit.

Add backup/restore plans #339

Add backup/restore plans #339

Conversation

timidri commented Feb 17, 2023 • edited by ragingra Loading

Summary

Checklist

Changes include test coverage?

Have you updated the documentation?

davidsandilands commented Mar 21, 2023

cdenneen commented Jun 29, 2023

Choose a reason for hiding this comment

timidri Nov 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bastelfreak Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bastelfreak Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timidri commented Feb 17, 2023 •

edited by ragingra

Loading

timidri Nov 21, 2023 •

edited

Loading

bastelfreak Jul 1, 2024 •

edited

Loading

bastelfreak Jul 1, 2024 •

edited

Loading