From d935b8d62ce58071786904c6d9e4ce477ab91e6f Mon Sep 17 00:00:00 2001 From: user202729 <25191436+user202729@users.noreply.github.com> Date: Sat, 14 Dec 2024 21:55:09 +0700 Subject: [PATCH 1/3] Add note about makeflags and ninja parallelism --- README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/README.md b/README.md index af91374fe19..f7844bf6ac7 100644 --- a/README.md +++ b/README.md @@ -341,6 +341,20 @@ in the Installation Guide. powerful machines, you might even consider `-j16`, as building with more jobs than CPU cores can speed things up further. + Alternatively, the `MAKEFLAGS` environment variable can be used. + In this case, only provide the flag itself, for example + `export MAKEFLAGS="-j4"`. + + Note that the compilation may nonetheless uses a different number of + threads, because sometimes `ninja` is used. + Unfortunately, [there is no way to control number of jobs `ninja` uses + from environment variables](https://github.com/ninja-build/ninja/issues/1482). + See also https://github.com/sagemath/sage/issues/38950. + + If the [Meson build system](https://doc-release--sagemath.netlify.app/html/en/installation/meson) + is used, the number of jobs running in parallel passed to `meson compile` will be respected, + because everything are managed by `ninja`. + To reduce the terminal output during the build, type `export V=0`. (`V` stands for "verbosity".) From d94fd8942587da666a730340bae75e00102d4d31 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20R=C3=BCth?= Date: Thu, 13 Feb 2025 15:26:42 +0100 Subject: [PATCH 2/3] Do not recommend to set MAKE=make -jX Setting MAKE=make -jX can lead to an exponential growth of build processes (as noted during SD128.) It's better to rely on make's jobserver that is available on macOS and all relevant Linux distributions. We also recommend a sane default for MAKEFLAGS that people can just copy & paste. Unfortunately, a lot of non-developers of SageMath still have to build from source and they appreciate not having to think about these things and just being able to copy a good default value. --- README.md | 32 +++++++++++------------------ src/doc/en/installation/source.rst | 33 ++++++++++++++++++------------ 2 files changed, 32 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 8a77ffe1b42..243e3c03868 100644 --- a/README.md +++ b/README.md @@ -334,26 +334,18 @@ in the Installation Guide. 11. Optional, but highly recommended: Set some environment variables to customize the build. - For example, the `MAKE` environment variable controls whether to - run several jobs in parallel. On a machine with 4 processors, say, - typing `export MAKE="make -j4"` will configure the build script to - perform a parallel compilation of Sage using 4 jobs. On some - powerful machines, you might even consider `-j16`, as building with - more jobs than CPU cores can speed things up further. - - Alternatively, the `MAKEFLAGS` environment variable can be used. - In this case, only provide the flag itself, for example - `export MAKEFLAGS="-j4"`. - - Note that the compilation may nonetheless uses a different number of - threads, because sometimes `ninja` is used. - Unfortunately, [there is no way to control number of jobs `ninja` uses - from environment variables](https://github.com/ninja-build/ninja/issues/1482). - See also https://github.com/sagemath/sage/issues/38950. - - If the [Meson build system](https://doc-release--sagemath.netlify.app/html/en/installation/meson) - is used, the number of jobs running in parallel passed to `meson compile` will be respected, - because everything are managed by `ninja`. + The `MAKEFLAGS` variable controls whether to run several jobs in parallel. + To saturate all the execution threads of your CPU, we recommend to run + `export MAKEFLAGS="-j$(nproc) -l$(nproc).5"` if you are on Linux, and + `export MAKEFLAGS="-j$(sysctl -n hw.ncpu) -l$(sysctl -n hw.ncpu).5"` if you + are on macOS. + + Note that the compilation may nonetheless use a different number of + processes, e.g., for parts that are built with `ninja` which automatically + decides on the amount of parallelity to use. In practice, you might + therefore see twice as many processes during the build process than your + CPU has execution threads. Unless your system is low on RAM, this should + not affect the time the compilation takes substantially. To reduce the terminal output during the build, type `export V=0`. (`V` stands for "verbosity".) diff --git a/src/doc/en/installation/source.rst b/src/doc/en/installation/source.rst index 12b1483dfa2..12ab938f0f3 100644 --- a/src/doc/en/installation/source.rst +++ b/src/doc/en/installation/source.rst @@ -669,8 +669,8 @@ Environment variables Sage uses several environment variables to control its build process. Most users won't need to set any of these: the build process just works on many platforms. -(Note though that setting :envvar:`MAKE`, as described below, can significantly -speed up the process.) +(Note though that setting :envvar:`MAKEFLAGS`, as described below, can +significantly speed up the process.) Building Sage involves building many packages, each of which has its own compilation instructions. @@ -680,19 +680,26 @@ Standard environment controlling the build process Here are some of the more commonly used variables affecting the build process: -.. envvar:: MAKE +.. envvar:: MAKEFLAGS - One useful setting for this variable when building Sage is - ``MAKE='make -jNUM'`` to tell the ``make`` program to run ``NUM`` jobs in - parallel when building. - Note that some Sage packages may not support this variable. + This variable can be set to tell the ``make`` program to build things in + parallel. Set it to ``-jNUM`` to run ``NUM`` jobs in parallel when building. + Add ``-lNUM`` to tell make not to spawn more processes when the load exceeds + ``NUM``. + + A good value for this variable is ``MAKEFLAGS="-j$(nproc) -l$(nproc).5"`` on + Linux and ``MAKEFLAGS="-j$(sysctl -n hw.ncpu) -l$(sysctl -n hw.ncpu).5"`` on + macOS. This instructs make to use all the execution threads of your CPU while + bounding the load if there are other processes generating load. If your + system does not have a lot of RAM, you might want to choose lower limits, if + you have lots of RAM, it can sometimes be beneficial to set these limits + slightly higher. + + Note that some parts of the SageMath build system do not respect this + variable, e.g., when ninja gets invoked, it figures out the number of + processes to use on its own so the number of processes and the system load + you see might exceed the number configured here. - Some people advise using more jobs than there are CPU cores, at least if the - system is not heavily loaded and has plenty of RAM; for example, a good - setting for ``NUM`` might be between 1 and 1.5 times the number of cores. - In addition, the ``-l`` option sets a load limit: ``MAKE='make -j4 -l5.5``, - for example, tells ``make`` to try to use four jobs, but to not start more - than one job if the system load average is above 5.5. See the manual page for GNU ``make``: `Command-line options `_ and `Parallel building From 2af2e49c2febb79559782003912d0cd7e535811b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Julian=20R=C3=BCth?= Date: Thu, 13 Feb 2025 16:20:18 +0100 Subject: [PATCH 3/3] Remove recommendation of MAKE from the FAQ --- src/doc/en/faq/faq-usage.rst | 2 +- src/doc/it/faq/faq-usage.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/doc/en/faq/faq-usage.rst b/src/doc/en/faq/faq-usage.rst index 393a9c369e5..ca7cab3458b 100644 --- a/src/doc/en/faq/faq-usage.rst +++ b/src/doc/en/faq/faq-usage.rst @@ -70,7 +70,7 @@ Sage. The command .. CODE-BLOCK:: shell-session - $ export MAKE='make -j8' + $ export MAKEFLAGS='-j8' will enable 8 threads for parts of the build that support parallelism. Change the number 8 as appropriate to suit the number of diff --git a/src/doc/it/faq/faq-usage.rst b/src/doc/it/faq/faq-usage.rst index 677d1a24bc2..2d2b32352a0 100644 --- a/src/doc/it/faq/faq-usage.rst +++ b/src/doc/it/faq/faq-usage.rst @@ -70,7 +70,7 @@ questi prerequisiti come segue:: Se hai un sistema multiprocessore puoi scegliere una compilazione parallela di Sage. Il comando :: - export MAKE='make -j8' + export MAKEFLAGS='-j8' abiliterà 8 threads per quelle parti della compilazione che supportano il parallelismo. Al posto del numero 8 metti il numero di