Make $pe_installer_source more flexible

This makes the peadm::install plan more flexible. Previously the
variable $pe_installer_source could point to an absolute URL directly to
the desired .tar.gz. Now it can also point to a web directory. Then
peadm will calculate the .tar.gz name and fetch it from the web
directory.

This applies for the peadm::upgrade and peadm::install plan.

Author: bastelfreak

REFERENCE.md

@@ -34,6 +34,7 @@
 * [`peadm::migration_opts_default`](#peadm--migration_opts_default)
 * [`peadm::node_manager_yaml_location`](#peadm--node_manager_yaml_location)
 * [`peadm::oid`](#peadm--oid)
+* [`peadm::pe_installer_source`](#peadm--pe_installer_source): calculates the PE installer URL and archive name
 * [`peadm::plan_step`](#peadm--plan_step)
 * [`peadm::recovery_opts_all`](#peadm--recovery_opts_all)
 * [`peadm::recovery_opts_default`](#peadm--recovery_opts_default)
@@ -805,6 +806,40 @@ Data type: `String`
 
 
 
+### <a name="peadm--pe_installer_source"></a>`peadm::pe_installer_source`
+
+Type: Puppet Language
+
+calculates the PE installer URL and archive name
+
+#### `peadm::pe_installer_source(Optional[String[1]] $pe_installer_source = undef, Optional[Peadm::Pe_version] $version = undef, Optional[String[1]] $platform = undef)`
+
+The peadm::pe_installer_source function.
+
+Returns: `Hash[String[1],String[1]]`
+
+##### `pe_installer_source`
+
+Data type: `Optional[String[1]]`
+
+The URL to download the Puppet Enterprise installer media from. If not
+specified, PEAdm will attempt to download PE installation media from its
+standard public source. When specified, PEAdm will download directly from the
+URL given. Can be an URL, that ends with a /, to a web directory that
+contains the original archives or an absolute URL to the .tar.gz archive.
+
+##### `version`
+
+Data type: `Optional[Peadm::Pe_version]`
+
+The desired version for PE. This is optional for custom provided absolute URLs.
+
+##### `platform`
+
+Data type: `Optional[String[1]]`
+
+The platform we're on, for example el-9-x86_64 (osfamily short name - version - arch)
+
 ### <a name="peadm--plan_step"></a>`peadm::plan_step`
 
 Type: Ruby 4.x API
@@ -1862,7 +1897,8 @@ Data type: `Optional[String]`
 The URL to download the Puppet Enterprise installer media from. If not
 specified, PEAdm will attempt to download PE installation media from its
 standard public source. When specified, PEAdm will download directly from the
-URL given.
+URL given. Can be an URL, that ends with a /, to a web directory that
+contains the original archives or an absolute URL to the .tar.gz archive.
 
 Default value: `undef`
 
@@ -2277,12 +2313,12 @@ The following parameters are available in the `peadm::upgrade` plan:
 * [`r10k_known_hosts`](#-peadm--upgrade--r10k_known_hosts)
 * [`stagingdir`](#-peadm--upgrade--stagingdir)
 * [`uploaddir`](#-peadm--upgrade--uploaddir)
+* [`version`](#-peadm--upgrade--version)
 * [`primary_host`](#-peadm--upgrade--primary_host)
 * [`replica_host`](#-peadm--upgrade--replica_host)
 * [`compiler_hosts`](#-peadm--upgrade--compiler_hosts)
 * [`primary_postgresql_host`](#-peadm--upgrade--primary_postgresql_host)
 * [`replica_postgresql_host`](#-peadm--upgrade--replica_postgresql_host)
-* [`version`](#-peadm--upgrade--version)
 * [`token_file`](#-peadm--upgrade--token_file)
 * [`download_mode`](#-peadm--upgrade--download_mode)
 * [`permit_unsafe_versions`](#-peadm--upgrade--permit_unsafe_versions)
@@ -2324,7 +2360,9 @@ Data type: `Optional[String]`
 The URL to download the Puppet Enterprise installer media from. If not
 specified, PEAdm will attempt to download PE installation media from its
 standard public source. When specified, PEAdm will download directly from the
-URL given.
+URL given. Can be an URL, that ends with a /, to a web directory that
+contains the original archives or an absolute URL to the .tar.gz archive.
+If it's an URL ending with the archive name, you don't need to set $version.
 
 Default value: `undef`
 
@@ -2366,6 +2404,14 @@ for offline usage.
 
 Default value: `'/tmp'`
 
+##### <a name="-peadm--upgrade--version"></a>`version`
+
+Data type: `Optional[Peadm::Pe_version]`
+
+The desired version for PE. This is optional for custom provided absolute URLs.
+
+Default value: `undef`
+
 ##### <a name="-peadm--upgrade--primary_host"></a>`primary_host`
 
 Data type: `Peadm::SingleTargetSpec`
@@ -2402,14 +2448,6 @@ Data type: `Optional[Peadm::SingleTargetSpec]`
 
 
 
-Default value: `undef`
-
-##### <a name="-peadm--upgrade--version"></a>`version`
-
-Data type: `Optional[Peadm::Pe_version]`
-
-
-
 Default value: `undef`
 
 ##### <a name="-peadm--upgrade--token_file"></a>`token_file`

functions/pe_installer_source.pp

@@ -0,0 +1,46 @@
+#
+# @summary calculates the PE installer URL and archive name
+#
+# @param pe_installer_source
+#   The URL to download the Puppet Enterprise installer media from. If not
+#   specified, PEAdm will attempt to download PE installation media from its
+#   standard public source. When specified, PEAdm will download directly from the
+#   URL given. Can be an URL, that ends with a /, to a web directory that
+#   contains the original archives or an absolute URL to the .tar.gz archive.
+#
+# @param version
+#  The desired version for PE. This is optional for custom provided absolute URLs.
+#
+# @param platform
+#  The platform we're on, for example el-9-x86_64 (osfamily short name - version - arch)
+#
+# @author Tim Meusel <[email protected]>
+#
+function peadm::pe_installer_source (
+  Optional[String[1]] $pe_installer_source = undef,
+  Optional[Peadm::Pe_version] $version     = undef,
+  Optional[String[1]] $platform            = undef,
+) >> Hash[String[1],String[1]] {
+  if $pe_installer_source {
+    # custom URL ends with /, so we assume it's a webdir with the original installer
+    if $pe_installer_source[-1] == '/' {
+      assert_type(Peadm::Pe_version, $version)
+      assert_type(String[1], $platform)
+      $_version          = $version
+      $pe_tarball_name   = "puppet-enterprise-${version}-${platform}.tar.gz"
+      $pe_tarball_source = "${pe_installer_source}${pe_tarball_name}"
+    } else {
+      $pe_tarball_name   = $pe_installer_source.split('/')[-1]
+      $pe_tarball_source = $pe_installer_source
+      $_version          = $pe_tarball_name.split('-')[2]
+    }
+    $data = { 'url' => $pe_tarball_source, 'filename' => $pe_tarball_name, 'version' => pick($_version,$version), }
+  } else {
+    assert_type(Peadm::Pe_version, $version)
+    assert_type(String[1], $platform)
+    $pe_tarball_name   = "puppet-enterprise-${version}-${platform}.tar.gz"
+    $pe_tarball_source = "https://s3.amazonaws.com/pe-builds/released/${version}/${pe_tarball_name}"
+    $data = { 'url' => $pe_tarball_source, 'filename' => $pe_tarball_name, 'version' => $version }
+  }
+  $data
+}

plans/install.pp

@@ -15,7 +15,8 @@
 #   The URL to download the Puppet Enterprise installer media from. If not
 #   specified, PEAdm will attempt to download PE installation media from its
 #   standard public source. When specified, PEAdm will download directly from the
-#   URL given.
+#   URL given. Can be an URL, that ends with a /, to a web directory that
+#   contains the original archives or an absolute URL to the .tar.gz archive.
 # @param ldap_config
 #   If specified, configures PE RBAC DS with the supplied configuration hash.
 #   The parameter should be set to a valid set of connection settings as

plans/subplans/install.pp

@@ -32,7 +32,8 @@
 #   The URL to download the Puppet Enterprise installer media from. If not
 #   specified, PEAdm will attempt to download PE installation media from its
 #   standard public source. When specified, PEAdm will download directly from the
-#   URL given.
+#   URL given. Can be an URL, that ends with a /, to a web directory that
+#   contains the original archives or an absolute URL to the .tar.gz archive.
 #
 plan peadm::subplans::install (
   # Standard
@@ -228,27 +229,21 @@
     )
   }
 
-  if $pe_installer_source {
-    $pe_tarball_name = $pe_installer_source.split('/')[-1]
-    $pe_tarball_source = $pe_installer_source
-  } else {
-    $pe_tarball_name   = "puppet-enterprise-${version}-${platform}.tar.gz"
-    $pe_tarball_source = "https://s3.amazonaws.com/pe-builds/released/${version}/${pe_tarball_name}"
-  }
+  $pe_installer = peadm::pe_installer_source($pe_installer_source, $version, $platform)
 
-  $upload_tarball_path = "${uploaddir}/${pe_tarball_name}"
+  $upload_tarball_path = "${uploaddir}/${pe_installer['filename']}"
 
   if $download_mode == 'bolthost' {
     # Download the PE tarball and send it to the nodes that need it
     run_plan('peadm::util::retrieve_and_upload', $pe_installer_targets,
-      source      => $pe_tarball_source,
-      local_path  => "${stagingdir}/${pe_tarball_name}",
+      source      => $pe_installer['url'],
+      local_path  => "${stagingdir}/${pe_installer['filename']}",
       upload_path => $upload_tarball_path,
     )
   } else {
     # Download PE tarballs directly to nodes that need it
     run_task('peadm::download', $pe_installer_targets,
-      source => $pe_tarball_source,
+      source => $pe_installer['url'],
       path   => $upload_tarball_path,
     )
   }

plans/upgrade.pp

@@ -15,7 +15,9 @@
 #   The URL to download the Puppet Enterprise installer media from. If not
 #   specified, PEAdm will attempt to download PE installation media from its
 #   standard public source. When specified, PEAdm will download directly from the
-#   URL given.
+#   URL given. Can be an URL, that ends with a /, to a web directory that
+#   contains the original archives or an absolute URL to the .tar.gz archive.
+#   If it's an URL ending with the archive name, you don't need to set $version.
 # @param final_agent_state
 #   Configures the state the puppet agent should be in on infrastructure nodes
 #   after PE is upgraded successfully.
@@ -30,6 +32,8 @@
 # @param uploaddir
 #   Directory the installer tarball will be uploaded to or expected to be in
 #   for offline usage.
+# @param version
+#  The desired version for PE. This is optional for custom provided absolute URLs.
 #
 plan peadm::upgrade (
   # Standard
@@ -117,21 +121,13 @@
 
   $platform = run_task('peadm::precheck', $primary_target).first['platform']
 
-  if $pe_installer_source {
-    $pe_tarball_name   = $pe_installer_source.split('/')[-1]
-    $pe_tarball_source = $pe_installer_source
-    $_version          = $pe_tarball_name.split('-')[2]
-  } else {
-    $_version          = $version
-    $pe_tarball_name   = "puppet-enterprise-${_version}-${platform}.tar.gz"
-    $pe_tarball_source = "https://s3.amazonaws.com/pe-builds/released/${_version}/${pe_tarball_name}"
-  }
+  $pe_installer = peadm::pe_installer_source($pe_installer_source, $version, $platform)
 
-  $upload_tarball_path = "${uploaddir}/${pe_tarball_name}"
+  $upload_tarball_path = "${uploaddir}/${pe_installer['filename']}"
 
   peadm::assert_supported_bolt_version()
 
-  peadm::assert_supported_pe_version($_version, $permit_unsafe_versions)
+  peadm::assert_supported_pe_version($pe_installer['version'], $permit_unsafe_versions)
 
   # Gather certificate extension information from all systems
   $cert_extensions = run_task('peadm::cert_data', $all_targets).reduce({}) |$memo,$result| {
@@ -176,14 +172,14 @@
     if $download_mode == 'bolthost' {
       # Download the PE tarball on the nodes that need it
       run_plan('peadm::util::retrieve_and_upload', $pe_installer_targets,
-        source      => $pe_tarball_source,
-        local_path  => "${stagingdir}/${pe_tarball_name}",
+        source      => $pe_installer['url'],
+        local_path  => "${stagingdir}/${pe_installer['filename']}",
         upload_path => $upload_tarball_path,
       )
     } else {
       # Download PE tarballs directly to nodes that need it
       run_task('peadm::download', $pe_installer_targets,
-        source => $pe_tarball_source,
+        source => $pe_installer['url'],
         path   => $upload_tarball_path,
       )
     }
@@ -418,7 +414,7 @@
     )
   }
 
-  peadm::check_version_and_known_hosts($current_pe_version, $_version, $r10k_known_hosts)
+  peadm::check_version_and_known_hosts($current_pe_version, $pe_installer['version'], $r10k_known_hosts)
 
   return("Upgrade of Puppet Enterprise ${arch['architecture']} completed.")
 }

spec/functions/pe_installer_source_spec.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+require 'spec_helper'
+
+describe 'peadm::pe_installer_source' do
+  it 'exists' do
+    is_expected.not_to be_nil
+  end
+
+  context 'when called with no parameters' do
+    it { is_expected.to run.with_params.and_raise_error(Puppet::PreformattedError) }
+  end
+  context 'when called with absolute url and version' do
+    result = {
+      'url'      => 'https://url/puppet-enterprise-2019.8.12-el-8-x86_64.tar.gz',
+      'filename' => 'puppet-enterprise-2019.8.12-el-8-x86_64.tar.gz',
+      'version'  => '2019.8.12'
+    }
+    it { is_expected.to run.with_params('https://url/puppet-enterprise-2019.8.12-el-8-x86_64.tar.gz', '2019.8.12').and_return(result) }
+  end
+  context 'when called with absolute url' do
+    result = {
+      'url'      => 'https://url/puppet-enterprise-2019.8.12-el-8-x86_64.tar.gz',
+      'filename' => 'puppet-enterprise-2019.8.12-el-8-x86_64.tar.gz',
+      'version'  => '2019.8.12'
+    }
+    it { is_expected.to run.with_params('https://url/puppet-enterprise-2019.8.12-el-8-x86_64.tar.gz').and_return(result) }
+  end
+  context 'when called with url and version and platform' do
+    result = {
+      'url'      => 'https://url/puppet-enterprise-2019.8.12-el-8-x86_64.tar.gz',
+      'filename' => 'puppet-enterprise-2019.8.12-el-8-x86_64.tar.gz',
+      'version'  => '2019.8.12'
+    }
+    it { is_expected.to run.with_params('https://url/', '2019.8.12', 'el-8-x86_64').and_return(result) }
+  end
+  context 'when called without url and with version and platform' do
+    result = {
+      'url'      => 'https://s3.amazonaws.com/pe-builds/released/2019.8.12/puppet-enterprise-2019.8.12-el-8-x86_64.tar.gz',
+      'filename' => 'puppet-enterprise-2019.8.12-el-8-x86_64.tar.gz',
+      'version'  => '2019.8.12'
+    }
+    it do
+      is_expected.to run.with_params(nil, '2019.8.12', 'el-8-x86_64').and_return(result)
+    end
+  end
+end

spec/plans/upgrade_spec.rb

@@ -28,7 +28,7 @@ def allow_standard_non_returning_calls
       .always_return({ 'content' => '2021.7.3' })
 
     expect_task('peadm::cert_data').return_for_targets('primary' => trusted_primary)
-
+    allow_task('peadm::precheck').return_for_targets('primary' => { 'hostname' => 'primary', 'platform' => 'el-7.11-x86_64' })
     expect(run_plan('peadm::upgrade',
                     'primary_host' => 'primary',
                     'version' => '2021.7.8')).to be_ok
@@ -43,6 +43,7 @@ def allow_standard_non_returning_calls
 
     expect_task('peadm::cert_data').return_for_targets('primary' => trusted_primary,
                                                        'compiler' => trusted_compiler)
+    allow_task('peadm::precheck').return_for_targets('primary' => { 'hostname' => 'primary', 'platform' => 'el-7.11-x86_64' })
 
     expect(run_plan('peadm::upgrade',
                     'primary_host' => 'primary',
@@ -92,6 +93,8 @@ def allow_standard_non_returning_calls
         .always_return({ 'content' => installed_version })
 
       expect_task('peadm::cert_data').return_for_targets('primary' => trusted_primary)
+
+      allow_task('peadm::precheck').return_for_targets('primary' => { 'hostname' => 'primary', 'platform' => 'el-7.11-x86_64' })
     end
 
     it 'updates pe.conf if r10k_known_hosts is set' do