lib-dynload installed in wrong location on a 64-bit system when CONFIG_SITE is set

[tfpf]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0a0 (heads/main:d4e534cbb3, Sep 2 2023, 21:58:58) [GCC 13.2.1 20230803 [revision cc279d6c64562f05019e1d12d0d825f9391b5553]]

A clear and concise description of the bug:

On openSUSE Tumbleweed 20230823, when CPython is compiled from source,

• lib-dynload is placed in /usr/local/lib64/python3.13.
• everything else is placed in /usr/local/lib/python3.13.

As a result, the interpreter is unable to locate some modules like readline (even though readline-devel is installed) and _posixsubprocess.

Here are the exact commands I ran.

$ CFLAGS='-march=native -mtune=native' ./configure --enable-optimizations --with-lto=yes
$ make -j4
$ sudo make -j4 altinstall
$ python3.13
Could not find platform dependent libraries <exec_prefix>
Python 3.13.0a0 (heads/main:d4e534cbb3, Sep  2 2023, 21:58:58) [GCC 13.2.1 20230803 [revision cc279d6c64562f05019e1d12d0d825f9391b5553]] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "/etc/pythonstart", line 7, in <module>
    import readline
ModuleNotFoundError: No module named 'readline'
>>>

Symlinking lib-dynload fixed the problem for me:

$ sudo ln -s /usr/local/lib64/python3.13/lib-dynload /usr/local/lib/python3.13
$ python3.13
Python 3.13.0a0 (heads/main:d4e534cbb3, Sep  2 2023, 21:58:58) [GCC 13.2.1 20230803 [revision cc279d6c64562f05019e1d12d0d825f9391b5553]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

but I believe everything should go in /usr/local/lib, as happens on other Linux distributions I have used.

Linked PRs

• GH-108819: fix LIBDEST not honoring --with-platlibdir #133163

[tfpf]

I tried to narrow down the source of the problem. According to this article, /usr/local/share/config.site is sourced if it exists. It does exist on my openSUSE Tumbleweed installation; sourcing it sets libdir to '${exec_prefix}/lib64'. Since CONFIG_SITE is set to /usr/share/site/x86_64-pc-linux-gnu, said file is sourced (this has the effect of setting libdir to '${exec_prefix}/lib64'), and the makefile generated contains, among other things:

LIBDIR=		${exec_prefix}/lib64
PLATLIBDIR=	lib

which I think is the reason lib-dynload went in the wrong location:

BINLIBDEST=	$(LIBDIR)/python$(VERSION)
DESTSHARED=	$(BINLIBDEST)/lib-dynload

while the remaining files, which depend on PLATLIBDIR (which is hard-coded in configure.ac) went in the expected location.

The solution is to clear CONFIG_SITE. When I built CPython using:

CONFIG_SITE="" ./configure
make -j4
sudo make -j4 altinstall

there were no errors. This mismatch between LIBDIR and PLATLIBDIR when CONFIG_SITE is set looks like a bug in the build process.

[tfpf]

I successfully reproduced the issue using Docker.

Dockerfile

FROM opensuse/tumbleweed
RUN zypper install -y gawk gcc git make zlib-devel
RUN git clone https://github.com/python/cpython.git

RUN mkdir -p /usr/share/site
COPY x86_64-pc-linux-gnu /usr/share/site
ENV CONFIG_SITE=/usr/share/site/x86_64-pc-linux-gnu

WORKDIR cpython
RUN ./configure
RUN make -j8
RUN make -j8 altinstall

CMD ["python3.13", "-c", "print(0)"]

x86_64-pc-linux-gnu (copied from my system)

#!/bin/sh
# Site script for configure. It is resourced via $CONFIG_SITE environment varaible.

# If user did not specify libdir, guess the correct target:
# Use lib64 for 64 bit bi-arch targets, keep the default for the rest.
if test "$libdir" = '${exec_prefix}/lib' ; then

	ac_config_site_64bit_host=NONE

	case "$host" in
	"" )
		# User did not specify host target.
		# The native platform x86_64 is a bi-arch platform.
		# Try to detect cross-compilation to inferior architecture.

		# We are trying to guess 32-bit target compilation. It's not as easy as
		# it sounds, as there is possible several intermediate combinations.
		ac_config_site_cross_to_32bit_host=NONE

		# User defined -m32 in CFLAGS or CXXFLAGS or CC or CXX:
		# (It's sufficient for 32-bit, but alone may cause mis-behavior of some checks.)
		case "$CFLAGS $CXXFLAGS $CC $CXX" in
		*-m32*)
			ac_config_site_cross_to_32bit_host=YES
			;;
		esac

		# Running with linux32:
		# (Changes detected platform, but not the toolchain target.)
		case "`/bin/uname -i`" in
		x86_64 | ppc64 | s390x | aarch64 )
			;;
		* )
			ac_config_site_cross_to_32bit_host=YES
			;;
		esac

		if test "x$ac_config_site_cross_to_32bit_host" = xNONE; then
			ac_config_site_64bit_host=YES
		fi

		;;
	*x86_64* | *ppc64* | *s390x* | *aarch64* )
		ac_config_site_64bit_host=YES
		;;
	esac

	if test "x$ac_config_site_64bit_host" = xYES; then
		libdir='${exec_prefix}/lib64'
	fi
fi

# Continue with the standard behavior of configure defined in AC_SITE_LOAD:
if test "x$prefix" != xNONE; then
	ac_site_files="$prefix/share/config.site $prefix/etc/config.site"
else
	ac_site_files="$ac_default_prefix/share/config.site $ac_default_prefix/etc/config.site"
fi

for ac_site_file in $ac_site_files
do
	case $ac_site_file in #(
	*/*) :
		 ;; #(
	*) :
		ac_site_file=./$ac_site_file ;;
esac
	if test -f "$ac_site_file" && test -r "$ac_site_file"; then
		{ printf "%s\n" "/usr/share/site/x86_64-pc-linux-gnu:${as_lineno-$LINENO}: loading site script $ac_site_file" >&5
printf "%s\n" "/usr/share/site/x86_64-pc-linux-gnu: loading site script $ac_site_file" >&6;}
		sed 's/^/| /' "$ac_site_file" >&5
		. "$ac_site_file" \
			|| { { printf "%s\n" "/usr/share/site/x86_64-pc-linux-gnu:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
printf "%s\n" "/usr/share/site/x86_64-pc-linux-gnu: error: in \`$ac_pwd':" >&2;}
as_fn_error $? "failed to load site script $ac_site_file
See \`config.log' for more details" "$LINENO" 5; }
	fi
done

docker build . -t gh-108819
docker run -t gh-108819

[slburson]

Hit the same problem installing 3.11.7 on openSUSE Leap 15.5.

[tfpf]

Could someone vet this? It looks like the solution would be to place everything in /usr/local/lib, ignoring CONFIG_SITE, but I'm guessing that'll have side-effects.

[johnny94]

I had the same problem when installing 3.12.4 on Rocky 8 & 9. You can add --with-platlibdir to specify platform-specific library directory. In my case, I use the following script to config building

./configure --prefix=/usr --enable-optimizations --with-platlibdir=lib64

Result

Python 3.12.4 (main, Aug  1 2024, 12:38:12) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/usr/lib64/python312.zip', '/usr/lib64/python3.12', '/usr/lib64/python3.12/lib-dynload', '/usr/lib64/python3.12/site-packages', '/usr/lib/python3.12/site-packages']
>>> print(sys.platlibdir)
lib64

Not necessary to create symlink for lib-dynload anymore.

[FFY00]

@tfpf, thank you for looking into this issue, I agree that your issue is likely due to a mismatch between the values used for libdir and platlibdir, which may be introduced by setting libdir to something other than ${exec_prefix}/lib in config.site.

If it isn't, that, at least, is an issue, given the current getpath.py code. I have opened GH-133163 with a fix for that, which I think should solve it on your side too.