Skip to content

Commit 58560cc

Browse files
committed
(PA-6422) Update rubygems patch for ruby 3.2.4
Previously, our rubygems patch wasn't being applied when cross compiling mac 12 ARM, because homebrew updated to ruby 3.2.4 and builder.rb was modified: - cmd = Gem.ruby.shellsplit + cmd = Shellwords.split(Gem.ruby) The code for patching is very fragile and has broken multiple times without any indication that the patch failed to apply, causing the resulting native extensions to be compiled for the wrong architecture. The code was also conflating the `settings[:ruby_version]` in our ruby component with the host's ruby version (the one we're actually trying to patch). Also the version that matters is the rubygems version (Gem::VERSION), not RUBY_VERSION. For example, if `gem update --system` was executed during provisioning, then our patch would fail to apply. This commit creates a script that runs on the remote system and so can introspect the host's ruby and rubygems versions. This way if brew updates to a newer ruby/rubygems, we can fail the build.
1 parent 63080c2 commit 58560cc

File tree

2 files changed

+162
-54
lines changed

2 files changed

+162
-54
lines changed

Diff for: configs/components/pl-ruby-patch.rb

+17-54
Original file line numberDiff line numberDiff line change
@@ -8,70 +8,33 @@
88
# This component should also be present in the puppet-agent project
99
component "pl-ruby-patch" do |pkg, settings, platform|
1010
if platform.is_cross_compiled?
11-
if platform.is_macos?
12-
pkg.build_requires 'gnu-sed'
13-
pkg.environment "PATH", "/usr/local/opt/gnu-sed/libexec/gnubin:$(PATH)"
14-
end
1511

16-
ruby_api_version = settings[:ruby_version].gsub(/\.\d*$/, '.0')
1712
ruby_version_y = settings[:ruby_version].gsub(/(\d+)\.(\d+)\.(\d+)/, '\1.\2')
1813

19-
base_ruby = case platform.name
20-
when /solaris-10/
21-
"/opt/csw/lib/ruby/2.0.0"
22-
when /osx/
23-
"/usr/local/opt/ruby@#{ruby_version_y}/lib/ruby/#{ruby_api_version}"
24-
else
25-
"/opt/pl-build-tools/lib/ruby/2.1.0"
26-
end
14+
pkg.add_source("file://resources/files/ruby/patch-hostruby.rb")
2715

2816
# The `target_triple` determines which directory native extensions are stored in the
2917
# compiled ruby and must match ruby's naming convention.
30-
#
31-
# solaris 10 uses ruby 2.0 which doesn't install native extensions based on architecture
32-
unless platform.name =~ /solaris-10/
33-
# weird architecture naming conventions...
34-
target_triple = if platform.architecture =~ /ppc64el|ppc64le/
35-
"powerpc64le-linux"
36-
elsif platform.name == 'solaris-11-sparc'
37-
"sparc-solaris-2.11"
38-
elsif platform.is_macos?
39-
if ruby_version_y.start_with?('2')
40-
"aarch64-darwin"
41-
else
42-
"arm64-darwin"
43-
end
18+
# weird architecture naming conventions...
19+
target_triple = if platform.architecture =~ /ppc64el|ppc64le/
20+
"powerpc64le-linux"
21+
elsif platform.name == 'solaris-11-sparc'
22+
"sparc-solaris-2.11"
23+
elsif platform.name =~ /solaris-10/
24+
"sparc-solaris"
25+
elsif platform.is_macos?
26+
if ruby_version_y.start_with?('2')
27+
"aarch64-darwin"
4428
else
45-
"#{platform.architecture}-linux"
29+
"arm64-darwin"
4630
end
31+
else
32+
"#{platform.architecture}-linux"
33+
end
4734

48-
pkg.build do
49-
[
50-
%(#{platform[:sed]} -i 's/Gem::Platform.local.to_s/"#{target_triple}"/' #{base_ruby}/rubygems/basic_specification.rb),
51-
%(#{platform[:sed]} -i 's/Gem.extension_api_version/"#{ruby_api_version}"/' #{base_ruby}/rubygems/basic_specification.rb)
52-
]
53-
end
54-
end
55-
56-
# make rubygems use our target rbconfig when installing gems
57-
case File.basename(base_ruby)
58-
when '2.0.0', '2.1.0'
59-
sed_command = %(s|Gem.ruby|&, '-r/opt/puppetlabs/puppet/share/doc/rbconfig-#{settings[:ruby_version]}-orig.rb'|)
60-
else
61-
sed_command = %(s|Gem.ruby.shellsplit|& << '-r/opt/puppetlabs/puppet/share/doc/rbconfig-#{settings[:ruby_version]}-orig.rb'|)
62-
end
63-
64-
# rubygems switched which file has the command we need to patch starting in rubygems 3.4.10, which we install in our formula
65-
# for ruby in homebrew-puppet
66-
if Gem::Version.new(settings[:ruby_version]) >= Gem::Version.new('3.2.2') || platform.is_macos? && ruby_version_y.start_with?('2')
67-
filename = 'builder.rb'
68-
else
69-
filename = 'ext_conf_builder.rb'
70-
end
71-
72-
pkg.build do
35+
pkg.install do
7336
[
74-
%(#{platform[:sed]} -i "#{sed_command}" #{base_ruby}/rubygems/ext/#{filename})
37+
"#{settings[:host_ruby]} patch-hostruby.rb #{settings[:ruby_version]} #{target_triple}"
7538
]
7639
end
7740
end

Diff for: resources/files/ruby/patch-hostruby.rb

+145
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# When cross compiling we need to run gem install using the host ruby, but
2+
# force ruby to use our overridden rbconfig.rb. To do that, we insert a
3+
# require statement between the ruby executable and it's first argument,
4+
# thereby hooking the ruby process.
5+
#
6+
# In the future we could use the --target-rbconfig=<path> option to point
7+
# to our rbconfig.rb. But that option is only available in newer ruby versions.
8+
require 'rbconfig'
9+
require 'tempfile'
10+
11+
if ARGV.length < 2
12+
warn <<USAGE
13+
USAGE: patch-hostruby.rb <target_ruby_version> <target_triple>
14+
15+
example: patch-hostruby.rb 3.2.2 arm64-darwin
16+
USAGE
17+
exit(1)
18+
end
19+
20+
# target ruby versions (what we're trying to build)
21+
target_ruby_version = ARGV[0]
22+
target_triple = ARGV[1]
23+
target_api_version = target_ruby_version.gsub(/\.\d*$/, '.0')
24+
25+
# host ruby (the ruby we execute to build the target)
26+
host_rubylibdir = RbConfig::CONFIG['rubylibdir']
27+
GEM_VERSION = Gem::Version.new(Gem::VERSION)
28+
29+
# Rewrite the file in-place securely, yielding each line to the caller
30+
def rewrite(file)
31+
# create temp file in the same directory as the file we're patching,
32+
# so rename doesn't cross filesystems
33+
tmpfile = Tempfile.new(File.basename(file), File.dirname(file))
34+
begin
35+
File.open("#{file}.orig", "w") do |orig|
36+
File.open(file, 'r').readlines.each do |line|
37+
orig.write(line)
38+
yield line
39+
tmpfile.write(line)
40+
end
41+
end
42+
ensure
43+
tmpfile.close
44+
File.unlink(file)
45+
File.rename(tmpfile.path, file)
46+
tmpfile.unlink
47+
end
48+
end
49+
50+
# Based on the RUBYGEMS version of the host ruby, the line and file that needs patching is different
51+
# Note the RUBY version doesn't matter (for either the host or target ruby).
52+
#
53+
# Here we define different intervals. For each interval, we specify the regexp to match, what to
54+
# replace it with, and which file to edit in-place. Note `\&` is a placeholder for whatever the regexp
55+
# was, that way we can easily append to it. And since it's in a double quoted string, it's escaped
56+
# as `\\&`
57+
#
58+
if GEM_VERSION <= Gem::Version.new('2.0.0')
59+
# $ git show v2.0.0:lib/rubygems/ext/ext_conf_builder.rb
60+
# cmd = "#{Gem.ruby} #{File.basename extension}"
61+
regexp = /{Gem\.ruby}/
62+
replace = "\\& -r/opt/puppetlabs/puppet/share/doc/rbconfig-#{target_ruby_version}-orig.rb"
63+
builder = 'rubygems/ext/ext_conf_builder.rb'
64+
elsif GEM_VERSION < Gem::Version.new('3.0.0') # there weren't any tags between >= 2.7.11 and < 3.0.0
65+
# $ git show v2.0.1:lib/rubygems/ext/ext_conf_builder.rb
66+
# cmd = [Gem.ruby, File.basename(extension), *args].join ' '
67+
#
68+
# $ git show v2.7.11:lib/rubygems/ext/ext_conf_builder.rb
69+
# cmd = [Gem.ruby, "-r", get_relative_path(siteconf.path), File.basename(extension), *args].join ' '
70+
regexp = /Gem\.ruby/
71+
replace = "\\&, '-r/opt/puppetlabs/puppet/share/doc/rbconfig-#{target_ruby_version}-orig.rb'"
72+
builder = 'rubygems/ext/ext_conf_builder.rb'
73+
elsif GEM_VERSION <= Gem::Version.new('3.4.8')
74+
# $ git show v3.0.0:lib/rubygems/ext/ext_conf_builder.rb
75+
# cmd = Gem.ruby.shellsplit << "-I" << File.expand_path("../../..", __FILE__) <<
76+
#
77+
# $ git show v3.4.8:lib/rubygems/ext/ext_conf_builder.rb
78+
# cmd = Gem.ruby.shellsplit << "-I" << File.expand_path("../..", __dir__) << File.basename(extension)
79+
regexp = /Gem\.ruby\.shellsplit/
80+
replace = "\\& << '-r/opt/puppetlabs/puppet/share/doc/rbconfig-#{target_ruby_version}-orig.rb'"
81+
builder = 'rubygems/ext/ext_conf_builder.rb'
82+
elsif GEM_VERSION <= Gem::Version.new('3.4.14')
83+
# NOTE: rubygems 3.4.9 moved the code to builder.rb
84+
#
85+
# $ git show v3.4.9:lib/rubygems/ext/builder.rb
86+
# cmd = Gem.ruby.shellsplit
87+
#
88+
# $ git show v3.4.14:lib/rubygems/ext/builder.rb
89+
# cmd = Gem.ruby.shellsplit
90+
regexp = /Gem\.ruby\.shellsplit/
91+
replace = "\\& << '-r/opt/puppetlabs/puppet/share/doc/rbconfig-#{target_ruby_version}-orig.rb'"
92+
builder = 'rubygems/ext/builder.rb'
93+
elsif GEM_VERSION <= Gem::Version.new('3.5.10')
94+
# $ git show v3.4.9:lib/rubygems/ext/builder.rb
95+
# cmd = Shellwords.split(Gem.ruby)
96+
#
97+
# $ git show v3.5.10:lib/rubygems/ext/builder.rb
98+
# cmd = Shellwords.split(Gem.ruby)
99+
regexp = /Shellwords\.split\(Gem\.ruby\)/
100+
replace = "\\& << '-r/opt/puppetlabs/puppet/share/doc/rbconfig-#{target_ruby_version}-orig.rb'"
101+
builder = 'rubygems/ext/builder.rb'
102+
else
103+
raise "We don't know how to patch rubygems #{GEM_VERSION}"
104+
end
105+
106+
# path to the builder file on the HOST ruby
107+
builder = File.join(host_rubylibdir, builder)
108+
109+
raise "We can't patch #{builder} because it doesn't exist" unless File.exist?(builder)
110+
111+
# hook rubygems builder so it loads our rbconfig when building native gems
112+
patched = false
113+
rewrite(builder) do |line|
114+
if line.gsub!(regexp, replace)
115+
patched = true
116+
end
117+
end
118+
119+
raise "Failed to patch rubygems hook, because we couldn't match #{regexp} in #{builder}" unless patched
120+
121+
puts "Patched '#{regexp.inspect}' in #{builder}"
122+
123+
# solaris 10 uses ruby 2.0 which doesn't install native extensions based on architecture
124+
if RUBY_PLATFORM !~ /solaris2\.10$/ || RUBY_VERSION != '2.0.0'
125+
# ensure native extensions are written to a directory that matches the
126+
# architecture of the target ruby we're building for. To do that we
127+
# patch the host ruby to pretend to be the target architecture.
128+
triple_patched = false
129+
api_version_patched = false
130+
spec_file = "#{host_rubylibdir}/rubygems/basic_specification.rb"
131+
rewrite(spec_file) do |line|
132+
if line.gsub!(/Gem::Platform\.local\.to_s/, "'#{target_triple}'")
133+
triple_patched = true
134+
end
135+
if line.gsub!(/Gem\.extension_api_version/, "'#{target_api_version}'")
136+
api_version_patched = true
137+
end
138+
end
139+
140+
raise "Failed to patch '#{target_triple}' in #{spec_file}" unless triple_patched
141+
puts "Patched '#{target_triple}' in #{spec_file}"
142+
143+
raise "Failed to patch '#{target_api_version}' in #{spec_file}" unless api_version_patched
144+
puts "Patched '#{target_api_version}' in #{spec_file}"
145+
end

0 commit comments

Comments
 (0)