Merge pull request #382 from puppetlabs/PE-36789

(PE-36789) R10k Known hosts upgrade path

Author: jpartlow

REFERENCE.md

@@ -20,6 +20,7 @@
 * [`peadm::assert_supported_pe_version`](#peadm--assert_supported_pe_version): Assert that the PE version given is supported by PEAdm
 * [`peadm::bolt_version`](#peadm--bolt_version)
 * [`peadm::certname`](#peadm--certname): Return the certname of the given target-like input
+* [`peadm::check_version_and_known_hosts`](#peadm--check_version_and_known_hosts): Checks PE verison and warns about setting r10k_known_hosts
 * [`peadm::convert_hash`](#peadm--convert_hash): converts two arrays into hash
 * [`peadm::convert_status`](#peadm--convert_status): Transforms a value in a human readable status with or without colors
 * [`peadm::determine_status`](#peadm--determine_status): Produces a summarized hash of the given status data
@@ -28,11 +29,13 @@
 * [`peadm::file_or_content`](#peadm--file_or_content)
 * [`peadm::flatten_compact`](#peadm--flatten_compact)
 * [`peadm::generate_pe_conf`](#peadm--generate_pe_conf): Generate a pe.conf file in JSON format
+* [`peadm::get_pe_conf`](#peadm--get_pe_conf)
 * [`peadm::get_targets`](#peadm--get_targets): Accept undef or a SingleTargetSpec, and return an Array[Target, 1, 0]. This differs from get_target() in that:   - It returns an Array[Target
 * [`peadm::node_manager_yaml_location`](#peadm--node_manager_yaml_location)
 * [`peadm::oid`](#peadm--oid)
 * [`peadm::plan_step`](#peadm--plan_step)
 * [`peadm::recovery_opts_default`](#peadm--recovery_opts_default)
+* [`peadm::update_pe_conf`](#peadm--update_pe_conf): Update the pe.conf file on a target with the provided hash
 * [`peadm::wait_until_service_ready`](#peadm--wait_until_service_ready): A convenience function to help remember port numbers for services and handle running the wait_until_service_ready task
 
 ### Data types
@@ -261,6 +264,56 @@ Variant[Target,
 
 
 
+### <a name="peadm--check_version_and_known_hosts"></a>`peadm::check_version_and_known_hosts`
+
+Type: Puppet Language
+
+Checks if the current PE version is less than 2023.3.0 and the target version is greater than or equal to 2023.3.0
+If both conditions are true and the r10k_known_hosts parameter is not defined, a warning message is displayed.
+
+#### `peadm::check_version_and_known_hosts(String $current_version, String $target_version, Optional[Peadm::Known_hosts] $r10k_known_hosts = undef)`
+
+Checks if the current PE version is less than 2023.3.0 and the target version is greater than or equal to 2023.3.0
+If both conditions are true and the r10k_known_hosts parameter is not defined, a warning message is displayed.
+
+Returns: `Any`
+
+##### `$current_version`
+
+Data type: `String`
+
+The current PE version
+
+##### `$target_version`
+
+Data type: `String`
+
+The target PE version
+
+##### `$r10k_known_hosts`
+
+Data type: `Optional[Peadm::Known_hosts]`
+
+The r10k_known_hosts parameter
+
+##### `current_version`
+
+Data type: `String`
+
+
+
+##### `target_version`
+
+Data type: `String`
+
+
+
+##### `r10k_known_hosts`
+
+Data type: `Optional[Peadm::Known_hosts]`
+
+
+
 ### <a name="peadm--convert_hash"></a>`peadm::convert_hash`
 
 Type: Puppet Language
@@ -653,6 +706,24 @@ Data type: `Hash`
 A hash of settings to set in the config file. Any keys that are set to
 undef will not be included in the config file.
 
+### <a name="peadm--get_pe_conf"></a>`peadm::get_pe_conf`
+
+Type: Puppet Language
+
+The peadm::get_pe_conf function.
+
+#### `peadm::get_pe_conf(Target $target)`
+
+The peadm::get_pe_conf function.
+
+Returns: `Any`
+
+##### `target`
+
+Data type: `Target`
+
+
+
 ### <a name="peadm--get_targets"></a>`peadm::get_targets`
 
 Type: Puppet Language
@@ -749,6 +820,30 @@ The peadm::recovery_opts_default function.
 
 Returns: `Any`
 
+### <a name="peadm--update_pe_conf"></a>`peadm::update_pe_conf`
+
+Type: Puppet Language
+
+Update the pe.conf file on a target with the provided hash
+
+#### `peadm::update_pe_conf(Target $target, Hash $updated_pe_conf_hash)`
+
+The peadm::update_pe_conf function.
+
+Returns: `Any`
+
+##### `target`
+
+Data type: `Target`
+
+The target to update the pe.conf file on
+
+##### `updated_pe_conf_hash`
+
+Data type: `Hash`
+
+The hash to update the pe.conf file with
+
 ### <a name="peadm--wait_until_service_ready"></a>`peadm::wait_until_service_ready`
 
 Type: Puppet Language
@@ -1932,6 +2027,7 @@ The following parameters are available in the `peadm::upgrade` plan:
 * [`internal_compiler_b_pool_address`](#-peadm--upgrade--internal_compiler_b_pool_address)
 * [`pe_installer_source`](#-peadm--upgrade--pe_installer_source)
 * [`final_agent_state`](#-peadm--upgrade--final_agent_state)
+* [`r10k_known_hosts`](#-peadm--upgrade--r10k_known_hosts)
 * [`primary_host`](#-peadm--upgrade--primary_host)
 * [`replica_host`](#-peadm--upgrade--replica_host)
 * [`compiler_hosts`](#-peadm--upgrade--compiler_hosts)
@@ -1993,6 +2089,17 @@ after PE is upgraded successfully.
 
 Default value: `'running'`
 
+##### <a name="-peadm--upgrade--r10k_known_hosts"></a>`r10k_known_hosts`
+
+Data type: `Optional[Peadm::Known_hosts]`
+
+Puppet Enterprise 2023.3+ requires host key verification for the
+r10k_remote host when using ssh. you must provide \$r10k_known_hosts
+information in the form of an array of hashes with 'name', 'type' and 'key'
+information for hostname, key-type and public key.
+
+Default value: `undef`
+
 ##### <a name="-peadm--upgrade--primary_host"></a>`primary_host`
 
 Data type: `Peadm::SingleTargetSpec`

documentation/upgrade.md

@@ -4,13 +4,15 @@ Puppet Enterprise deployments provisioned using the peadm module can also be upg
 
 ## Usage
 
-The `peadm::upgrade` plan requires as input the version of PE to upgrade to, and the names of each PE infrastructure host. Primary, replica, compilers, etc.
+The `peadm::upgrade` plan requires as input the version of PE to upgrade to, and the names of each PE infrastructure host. Primary, replica, compilers, etc. 
 
-The following is an example parameters file for upgrading an Extra Large architecture deployment of PE 2021.0.1 to PE 2021.7.4.
+Please note that when upgrading from before 2023.3 to 2023.3 or above and you are using code manager, it is nessesary to provide known hosts for r10k. r10k_known_hosts is an optional parameter and is only required one time when upgrading to 2023.3 or beyond. But if you currently use the SSH protocol to allow r10k to access your remote Git repository, your Code manager or r10k code management tool cannot function until you define the r10k_known_hosts parameter. Subsequent upgrades will already have this and it won't be required again. Please refer to the Puppet Enterprise 2023.3 Upgrade cautions for more details.
+
+The following is an example parameters file for upgrading an Extra Large architecture deployment of PE 2023.2.0 to PE 2023.3.0.
 
 ```json
 {
-  "version": "2021.7.4",
+  "version": "2023.3.0",
   "primary_host": "pe-master-09a40c-0.us-west1-a.c.reidmv-peadm.internal",
   "primary_postgresql_host": "pe-psql-09a40c-0.us-west1-a.c.reidmv-peadm.internal",
   "replica_host": "pe-master-09a40c-1.us-west1-b.c.reidmv-peadm.internal",
@@ -20,6 +22,10 @@ The following is an example parameters file for upgrading an Extra Large archite
     "pe-compiler-09a40c-1.us-west1-b.c.reidmv-peadm.internal",
     "pe-compiler-09a40c-2.us-west1-c.c.reidmv-peadm.internal",
     "pe-compiler-09a40c-3.us-west1-a.c.reidmv-peadm.internal"
+  ],
+  "r10k_known_hosts": [
+     {"name": "remotehostname", "type": "ssh-rsa", "key": "hash"},
+     {"name": "remotehostname2", "type": "ssh-rsa", "key": "hash"}
   ]
 }
 ```
@@ -115,20 +121,20 @@ Note: it is assumed that the Puppet primary is in cluster A when the upgrade sta
 1. Shut down the `pe-puppetdb` service on the compilers in cluster B
 2. If different from the primary (replica), run the `install-puppet-enterprise` script for the new PE version on the PuppetDB PostgreSQL node for cluster B
 3. If different from the primary (replica), Run `puppet agent -t` on the PuppetDB PostgreSQL node for cluster B
-5. Run `puppet agent -t` on the primary to ensure orchestration services are configured and restarted before the next steps
-6. Perform the replica upgrade using `puppet infra upgrade replica` for the primary (replica)
-7. Perform the compiler upgrade using `puppet infra upgrade compiler` for the compilers in cluster B
+4. Run `puppet agent -t` on the primary to ensure orchestration services are configured and restarted before the next steps
+5. Perform the replica upgrade using `puppet infra upgrade replica` for the primary (replica)
+6. Perform the compiler upgrade using `puppet infra upgrade compiler` for the compilers in cluster B
 
 **If Upgrading from 2019.5**
 
 The following steps apply _only_ if upgrading from 2019.5 or older
 
 1. Run `puppet infra run convert_legacy_compiler` for all compilers
 2. Modify the peadm node groups "PE Compiler Group A" and "PE Compiler Group B" as follows:
-    * Re-parent the groups. They should be children of "PE Compiler"
-    * Remove configuration data (Hiera data). Leave the classes and class parameters
-    * Add the rule `trusted.extensions.pp_auth_role = pe_compiler`
-    * Remove the rule `trusted.extensions."1.3.6.1.4.1.34380.1.1.9812" = puppet/compiler`
+   * Re-parent the groups. They should be children of "PE Compiler"
+   * Remove configuration data (Hiera data). Leave the classes and class parameters
+   * Add the rule `trusted.extensions.pp_auth_role = pe_compiler`
+   * Remove the rule `trusted.extensions."1.3.6.1.4.1.34380.1.1.9812" = puppet/compiler`
 
 **Phase 4: resume puppet service**
 

examples/upgrade.json

@@ -0,0 +1,17 @@
+{
+    "version": "2023.3.0",
+    "primary_host": "pe-master-09a40c-0.us-west1-a.c.reidmv-peadm.internal",
+    "primary_postgresql_host": "pe-psql-09a40c-0.us-west1-a.c.reidmv-peadm.internal",
+    "replica_host": "pe-master-09a40c-1.us-west1-b.c.reidmv-peadm.internal",
+    "replica_postgresql_host": "pe-psql-09a40c-1.us-west1-b.c.reidmv-peadm.internal",
+    "compiler_hosts": [
+        "pe-compiler-09a40c-0.us-west1-a.c.reidmv-peadm.internal",
+        "pe-compiler-09a40c-1.us-west1-b.c.reidmv-peadm.internal",
+        "pe-compiler-09a40c-2.us-west1-c.c.reidmv-peadm.internal",
+        "pe-compiler-09a40c-3.us-west1-a.c.reidmv-peadm.internal"
+    ],
+    "r10k_known_hosts": [
+        {"name": "remotehostname", "type": "ssh-rsa", "key": "hash"},
+        {"name": "remotehostname2", "type": "ssh-rsa", "key": "hash"}
+    ]
+}

functions/check_version_and_known_hosts.pp

@@ -0,0 +1,26 @@
+# @summary Checks PE verison and warns about setting r10k_known_hosts
+# Checks if the current PE version is less than 2023.3.0 and the target version is greater than or equal to 2023.3.0
+# If both conditions are true and the r10k_known_hosts parameter is not defined, a warning message is displayed.
+# @param $current_version [String] The current PE version
+# @param $target_version [String] The target PE version
+# @param $r10k_known_hosts [Optional[Peadm::Known_hosts]] The r10k_known_hosts parameter
+function peadm::check_version_and_known_hosts(
+  String $current_version,
+  String $target_version,
+  Optional[Peadm::Known_hosts]      $r10k_known_hosts         = undef,
+) {
+  $version = '2023.3.0'
+  $current_check = SemVer($current_version) < SemVer($version)
+  $target_check = SemVer($target_version) >= SemVer($version)
+
+  # lint:ignore:140chars
+  if ($current_check and $target_check and $r10k_known_hosts == undef) {
+    out::message( @(HEREDOC/n)
+\nWARNING: Starting in PE 2023.3, SSH host key verification is required for Code Manager and r10k.\n
+To enable host key verification, you must define the puppet_enterprise::profile::master::r10k_known_hosts parameter with an array of hashes containing "name", "type", and "key" to specify your hostname, key type, and public key for your remote host(s).\n
+If you currently use SSH protocol to allow r10k to access your remote Git repository, your Code Manager or r10k code management tool cannot function until you define the r10k_known_hosts parameter.\n
+Please refer to the Puppet Enterprise 2023.3 Upgrade cautions for more details.\n
+HEREDOC
+    )# lint:endignore
+  }
+}

functions/get_pe_conf.pp

@@ -0,0 +1,9 @@
+function peadm::get_pe_conf(Target $target) {
+  $current_pe_conf_content = run_task('peadm::read_file', $target, path => '/etc/puppetlabs/enterprise/conf.d/pe.conf').first['content']
+
+  # Parse the current pe.conf content and return the hash
+  return $current_pe_conf_content ? {
+    undef   => {},
+    default => stdlib::parsehocon($current_pe_conf_content),
+  }
+}

functions/update_pe_conf.pp

@@ -0,0 +1,10 @@
+# @summary Update the pe.conf file on a target with the provided hash
+# @param target [Bolt::Target] The target to update the pe.conf file on
+# @param updated_pe_conf_hash [Hash] The hash to update the pe.conf file with
+function peadm::update_pe_conf(Target $target, Hash $updated_pe_conf_hash) {
+  # Convert the updated hash back to a pretty JSON string
+  $updated_pe_conf_content = stdlib::to_json_pretty($updated_pe_conf_hash)
+
+  # Write the updated content back to pe.conf on the target
+  write_file($updated_pe_conf_content, '/etc/puppetlabs/enterprise/conf.d/pe.conf', $target)
+}

plans/upgrade.pp

@@ -19,7 +19,12 @@
 # @param final_agent_state
 #   Configures the state the puppet agent should be in on infrastructure nodes
 #   after PE is upgraded successfully.
-#
+# @param r10k_known_hosts
+#   Puppet Enterprise 2023.3+ requires host key verification for the
+#   r10k_remote host when using ssh. you must provide \$r10k_known_hosts
+#   information in the form of an array of hashes with 'name', 'type' and 'key'
+#   information for hostname, key-type and public key.
+# 
 plan peadm::upgrade (
   # Standard
   Peadm::SingleTargetSpec           $primary_host,
@@ -33,11 +38,12 @@
   Optional[Peadm::SingleTargetSpec] $replica_postgresql_host = undef,
 
   # Common Configuration
-  Optional[Peadm::Pe_version] $version                          = undef,
-  Optional[String]            $pe_installer_source              = undef,
-  Optional[String]            $compiler_pool_address            = undef,
-  Optional[String]            $internal_compiler_a_pool_address = undef,
-  Optional[String]            $internal_compiler_b_pool_address = undef,
+  Optional[Peadm::Pe_version]  $version                          = undef,
+  Optional[String]             $pe_installer_source              = undef,
+  Optional[String]             $compiler_pool_address            = undef,
+  Optional[String]             $internal_compiler_a_pool_address = undef,
+  Optional[String]             $internal_compiler_b_pool_address = undef,
+  Optional[Peadm::Known_hosts] $r10k_known_hosts                 = undef,
 
   # Other
   Optional[String]           $token_file             = undef,
@@ -210,6 +216,17 @@
 
       write_file($pe_conf, '/etc/puppetlabs/enterprise/conf.d/pe.conf', $target)
     }
+
+    if $r10k_known_hosts != undef {
+      $current_pe_conf = peadm::get_pe_conf($primary_target[0])
+
+      # Append the r10k_known_hosts entry
+      $updated_pe_conf = $current_pe_conf + {
+        'puppet_enterprise::profile::master::r10k_known_hosts' => $r10k_known_hosts,
+      }
+
+      peadm::update_pe_conf($primary_target[0], $updated_pe_conf)
+    }
   }
 
   peadm::plan_step('upgrade-primary') || {
@@ -391,5 +408,7 @@
     )
   }
 
+  peadm::check_version_and_known_hosts($current_pe_version, $_version, $r10k_known_hosts)
+
   return("Upgrade of Puppet Enterprise ${arch['architecture']} completed.")
 }

spec/plans/upgrade_spec.rb

@@ -23,7 +23,10 @@ def allow_standard_non_returning_calls
   it 'minimum variables to run' do
     allow_standard_non_returning_calls
 
-    expect_task('peadm::read_file').always_return({ 'content' => 'mock' })
+    expect_task('peadm::read_file')
+      .with_params('path' => '/opt/puppetlabs/server/pe_build')
+      .always_return({ 'content' => '2021.7.3' })
+
     expect_task('peadm::cert_data').return_for_targets('primary' => trusted_primary)
 
     expect(run_plan('peadm::upgrade',
@@ -34,7 +37,10 @@ def allow_standard_non_returning_calls
   it 'runs with a primary, compilers, but no replica' do
     allow_standard_non_returning_calls
 
-    expect_task('peadm::read_file').always_return({ 'content' => 'mock' })
+    expect_task('peadm::read_file')
+      .with_params('path' => '/opt/puppetlabs/server/pe_build')
+      .always_return({ 'content' => '2021.7.3' })
+
     expect_task('peadm::cert_data').return_for_targets('primary' => trusted_primary,
                                                        'compiler' => trusted_compiler)
 
@@ -55,4 +61,80 @@ def allow_standard_non_returning_calls
     expect(result.value.kind).to eq('unexpected-transport')
     expect(result.value.msg).to match(%r{The "pcp" transport is not available for use with the Primary})
   end
+
+  context 'r10k_known_hosts' do
+    let(:installed_version) { '2021.7.3' }
+    let(:r10k_known_hosts) do
+      [
+        {
+          'name'          => 'primary.rspec',
+          'type'          => 'rsa',
+          'key'           => 'pubkey',
+        },
+      ]
+    end
+    # NOTE: dupliating this error message is unfortunate, but
+    # expect_out_message() doesn't take a regex.
+    let(:r10k_warning) do
+      <<~EOS
+        \nWARNING: Starting in PE 2023.3, SSH host key verification is required for Code Manager and r10k.\n
+        To enable host key verification, you must define the puppet_enterprise::profile::master::r10k_known_hosts parameter with an array of hashes containing "name", "type", and "key" to specify your hostname, key type, and public key for your remote host(s).\n
+        If you currently use SSH protocol to allow r10k to access your remote Git repository, your Code Manager or r10k code management tool cannot function until you define the r10k_known_hosts parameter.\n
+        Please refer to the Puppet Enterprise 2023.3 Upgrade cautions for more details.\n
+      EOS
+    end
+
+    before(:each) do
+      allow_standard_non_returning_calls
+
+      expect_task('peadm::read_file')
+        .with_params('path' => '/opt/puppetlabs/server/pe_build')
+        .always_return({ 'content' => installed_version })
+
+      expect_task('peadm::cert_data').return_for_targets('primary' => trusted_primary)
+    end
+
+    it 'updates pe.conf if r10k_known_hosts is set' do
+      expect_task('peadm::read_file')
+        .with_params('path' => '/etc/puppetlabs/enterprise/conf.d/pe.conf')
+        .always_return({ 'content' => <<~PECONF })
+          # spec pe.conf
+          "puppet_enterprise::puppet_master_host": "%{::trusted.certname}"
+        PECONF
+      # TODO: this doesn't verify what we are writing; we would need to mock
+      # write_file for that. Being more specific about exactly what file we are
+      # uploading runs afoul of the fact that write_file creates a source tempfile,
+      # and we can't expect_upload() because we don't have the tempfile name.
+      allow_any_upload
+
+      expect(run_plan('peadm::upgrade',
+                       'primary_host'     => 'primary',
+                       'version'          => '2023.3.0',
+                       'r10k_known_hosts' => r10k_known_hosts,
+                       'permit_unsafe_versions' => true)).to be_ok
+    end
+
+    it 'warns if upgrading to 2023.3+ from 2023.0- without r10k_known_hosts set' do
+      # This is fairly horrible, but expect_out_message doesn't take a regex.
+      expect_out_message.with_params(r10k_warning)
+
+      expect(run_plan('peadm::upgrade',
+                       'primary_host'     => 'primary',
+                       'version'          => '2023.3.0',
+                       'permit_unsafe_versions' => true)).to be_ok
+    end
+
+    context 'upgrading from 2023.3+' do
+      let(:installed_version) { '2023.3.0' }
+
+      it 'does not warn if r10k_known_hosts is not set' do
+        expect_out_message.with_params(r10k_warning).not_be_called
+
+        expect(run_plan('peadm::upgrade',
+                         'primary_host'     => 'primary',
+                         'version'          => '2023.4.0',
+                         'permit_unsafe_versions' => true)).to be_ok
+      end
+    end
+  end
 end