Skip to content

Commit cec99ed

Browse files
gh-89792: Limit test_tools freeze test build parallelism based on the number of cores (GH-101841)
unhardcode freeze test build parallelism. base it on the number of cpus, don't use more than max(2, os.cpu_count()/3). (cherry picked from commit dfc2e06) Co-authored-by: Gregory P. Smith <[email protected]>
1 parent d17cc3d commit cec99ed

File tree

2 files changed

+16
-6
lines changed

2 files changed

+16
-6
lines changed
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1-
``test_tools`` now copies up to 10x less source data to a temporary
2-
directory during the ``freeze`` test by ignoring git metadata and other
3-
artifacts.
1+
``test_tools`` now copies up to 10x less source data to a temporary directory
2+
during the ``freeze`` test by ignoring git metadata and other artifacts. It
3+
also limits its python build parallelism based on os.cpu_count instead of hard
4+
coding it as 8 cores.

Tools/freeze/test/freeze.py

+12-3
Original file line numberDiff line numberDiff line change
@@ -163,16 +163,25 @@ def prepare(script=None, outdir=None):
163163
if not MAKE:
164164
raise UnsupportedError('make')
165165

166+
cores = os.cpu_count()
167+
if cores and cores >= 3:
168+
# this test is most often run as part of the whole suite with a lot
169+
# of other tests running in parallel, from 1-2 vCPU systems up to
170+
# people's NNN core beasts. Don't attempt to use it all.
171+
parallel = f'-j{cores*2//3}'
172+
else:
173+
parallel = '-j2'
174+
166175
# Build python.
167-
print(f'building python in {builddir}...')
176+
print(f'building python {parallel=} in {builddir}...')
168177
if os.path.exists(os.path.join(srcdir, 'Makefile')):
169178
# Out-of-tree builds require a clean srcdir.
170179
_run_quiet([MAKE, '-C', srcdir, 'clean'])
171-
_run_quiet([MAKE, '-C', builddir, '-j8'])
180+
_run_quiet([MAKE, '-C', builddir, parallel])
172181

173182
# Install the build.
174183
print(f'installing python into {prefix}...')
175-
_run_quiet([MAKE, '-C', builddir, '-j8', 'install'])
184+
_run_quiet([MAKE, '-C', builddir, 'install'])
176185
python = os.path.join(prefix, 'bin', 'python3')
177186

178187
return outdir, scriptfile, python

0 commit comments

Comments
 (0)