Envvars not set in subshell in macOS Python 3.7-3.9 configure invocation

[cesarcoatl]

Describe the bug

I've tried building Python 3.9.16 on macOS following the dev guide and I believe there is an issue with the command as pkg-config fails to get the value set for PKG_CONFIG_PATH as, in my limited knowledge of the shell, I believe pkg-config --libs tcl tk runs on a subshell.

The current configure command produces the following output:

$ CFLAGS="-I$(brew --prefix gdbm)/include -I$(brew --prefix xz)/include" \
  LDFLAGS="-L$(brew --prefix gdbm)/lib -L$(brew --prefix xz)/lib" \
  PKG_CONFIG_PATH="$(brew --prefix tcl-tk)/lib/pkgconfig" \
  ./configure --with-pydebug \
          --with-openssl=$(brew --prefix [email protected]) \
          --with-tcltk-libs="$(pkg-config --libs tcl tk)" \
          --with-tcltk-includes="$(pkg-config --cflags tcl tk)"
Package tcl was not found in the pkg-config search path.
Perhaps you should add the directory containing `tcl.pc'
to the PKG_CONFIG_PATH environment variable
No package 'tcl' found
Package tk was not found in the pkg-config search path.
Perhaps you should add the directory containing `tk.pc'
to the PKG_CONFIG_PATH environment variable
No package 'tk' found
Package tcl was not found in the pkg-config search path.
Perhaps you should add the directory containing `tcl.pc'
to the PKG_CONFIG_PATH environment variable
No package 'tcl' found
Package tk was not found in the pkg-config search path.
Perhaps you should add the directory containing `tk.pc'
to the PKG_CONFIG_PATH environment variable
No package 'tk' found

Expected behavior

The one-line command should properly set PKG_CONFIG_PATH.

I've had success running the following:

$ export PKG_CONFIG_PATH="$(brew --prefix tcl-tk)/lib/pkgconfig"; \
CFLAGS="-I$(brew --prefix gdbm)/include -I$(brew --prefix xz)/include" \
LDFLAGS="-L$(brew --prefix gdbm)/lib -L$(brew --prefix xz)/lib" \
./configure --with-pydebug \
          --with-openssl=$(brew --prefix [email protected]) \
          --with-tcltk-libs="$(pkg-config --libs tcl tk)" \
          --with-tcltk-includes="$(pkg-config --cflags tcl tk)"
checking build system type... aarch64-apple-darwin22.3.0
checking host system type... aarch64-apple-darwin22.3.0
checking for python3.9... python3.9
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... "darwin"
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for a sed that does not truncate output... /usr/bin/sed
checking for --with-cxx-main=<compiler>... no
checking for g++... no
configure:

  By default, distutils will build C++ extension modules with "g++".
  If this is not intended, then set CXX on the configure command line.

checking for the platform triplet based on compiler characteristics... darwin
checking for multiarch...
checking for -Wl,--no-as-needed... no
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... ^C%

Additional context

There must be something that can be done differently, but I know I'm no expert in the matter.

[CAM-Gerlach]

Does anything need to change in the devguide documentation, or are you suggesting the generated configure script should be tweaked? If the former, could you point out exactly where the text should be changed is, and what you're suggesting it be changed to? If the latter, we should transfer this issue to the CPython repo instead.

[cesarcoatl]

Hello, @CAM-Gerlach. Thanks for following up on this.

I am suggesting a change to the instructions for building Python on macOS here.

The command for building Python 3.9 through 3.7 is not working as intended because pkg-config cannot find tcl and tk.

On my comment before yours I've posted the command that worked:

$ export PKG_CONFIG_PATH="$(brew --prefix tcl-tk)/lib/pkgconfig"; \
CFLAGS="-I$(brew --prefix gdbm)/include -I$(brew --prefix xz)/include" \
LDFLAGS="-L$(brew --prefix gdbm)/lib -L$(brew --prefix xz)/lib" \
./configure --with-pydebug \
          --with-openssl=$(brew --prefix [email protected]) \
          --with-tcltk-libs="$(pkg-config --libs tcl tk)" \
          --with-tcltk-includes="$(pkg-config --cflags tcl tk)"

But I would like to open this up for discussion as there might be a better approach since I am no expert on that matter.

[CAM-Gerlach]

Okay, thanks for the clarifications. In that case, I'd refer this to macOS build experts @ned-deily and @ronaldoussoren to take a look.

[erlend-aasland]

@thecesrom, can you please propose a patch in the form of a PR; it's easier to review your proposed change that way.

[ned-deily]

I don't use Homebrew myself but the problem here is a shell issue. It's a bit subtle but the instructions for both 3.9- and 3.10+ are incorrect in the devguide. The issue is the order in which a POSIX-conformant shell evaluates statements. Reading the POSIX spec for the shell command language, it states that, when evaluating simple commands, words that are recognized as variable assignments are saved for processing in a later step, then words (in the shell sense) are expanded first, including command substitions $( ), and then variable assignments are processed. So, at the time the --with-tcltk-libs="$(pkg-config --libs tcl tk)" is expanded, the PKG_CONFIG_PATH variable assignment hasn't happened yet. A simpler example shows this:

PKG_CONFIG_PATH="$(brew --prefix tcl-tk)/lib/pkgconfig"  echo $(pkg-config --libs tcl tk)
Package tcl was not found in the pkg-config search path.
Perhaps you should add the directory containing `tcl.pc'
to the PKG_CONFIG_PATH environment variable
No package 'tcl' found
Package tk was not found in the pkg-config search path.
Perhaps you should add the directory containing `tk.pc'
to the PKG_CONFIG_PATH environment variable
No package 'tk' found

For the 3.10 instructions, it doesn't matter that the PKG_CONFIG_PATH definition doesn't work because configure.ac was enhanced in 3.10 to call pkg-config directly.

And things are much simplier starting with 3.11 as configure.ac and friends almost fully support pkg-config for all dependencies. The devguide should be updated to reflect that.

[cesarcoatl]

@thecesrom, can you please propose a patch in the form of a PR; it's easier to review your proposed change that way.

Will do.

[CAM-Gerlach]

And @thecesrom make sure to see Ned's comment for what actually should be changed before you do.

[cesarcoatl]

If I understood @ned-deily's comment correctly, it is not necessary to set PKG_CONFIG_PATH.

So I've just tried it with this:

$ CFLAGS="-I$(brew --prefix gdbm)/include -I$(brew --prefix xz)/include" \
  LDFLAGS="-L$(brew --prefix gdbm)/lib -I$(brew --prefix xz)/lib" \
  ./configure --with-pydebug \
              --with-openssl=$(brew --prefix openssl)

And while I did get some warnings after running make, this is the the output

$ make -s -j2
...
The necessary bits to build these optional modules were not found:
ossaudiodev           spwd
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  pwd                   time

renaming build/scripts-3.10/pydoc3 to build/scripts-3.10/pydoc3.10
renaming build/scripts-3.10/idle3 to build/scripts-3.10/idle3.10
renaming build/scripts-3.10/2to3 to build/scripts-3.10/2to3-3.10

I do have one more suggestion for the brew install command, as the guide is only suggesting [email protected] while the command refers to openssl.

The PR is on its way.

Thanks