handling multiple pythons

[charlesreid1]

Just throwing an idea out there, as far as handling arbitrary python versions - you could install pyenv in the container, which would give access to a really nice version management system (and not just python, but pypy, anaconda, and friends). You could put pyenv in the container itself, or just use it to grab the right version of python during the build process. What do you think?

It also looks like the official python alpine images you linked to in the readme (link is broken now) provide multiple 3.x versions of python: https://github.com/docker-library/python - but that's much less fine-grained than pyenv.

[jfloff]

Mmm that's interesting. Would you be interested in doing a couple of tests and get back with some numbers? Mainly what is the impact on container size.

[jfloff]

Hey @charlesreid1,

Also pinging @tprobinson for feedback

I was able to test this, and after a couple of optimizations regarding size (deleting everything that seemed unnecessary), I was able to reach pretty good results. At the end of the day, I was able to add pyenv which in fact enables us to install any version of python on-demand, with just a 4MB increase on size (I was testing on python 2.7, I guess other versions would get the same overhead).

Could you please take a look and please tell me if anything can be further optimized regarding size? Here is the full Dockerfile for python 2.7:

FROM alpine:3.7

# VERSIONS
ENV ALPINE_VERSION=3.7 \
    PYTHON_VERSION=2.7.14

# PATHS
ENV PYTHON_PATH=/usr/lib/python$PYTHON_VERSION \
    PATH="/usr/lib/python$PYTHON_VERSION/bin/:${PATH}"

# PACKAGES
#   * dumb-init: a proper init system for containers, to reap zombie children
#   * musl: standard C library
#   * lib6-compat: compatibility libraries for glibc
#   * linux-headers: commonly needed, and an unusual package name from Alpine.
#   * build-base: used so we include the basic development packages (gcc)
#   * bash: so we can access /bin/bash
#   * git: to ease up clones of repos
#   * ca-certificates: for SSL verification during Pip and easy_install
ENV PACKAGES="\
    dumb-init \
    musl \
    libc6-compat \
    linux-headers \
    build-base \
    bash \
    git \
    ca-certificates \
"

# PACKAGES needed to built python
ENV PYTHON_BUILD_PACKAGES="\
    readline-dev \
    zlib-dev \
    bzip2-dev \
    sqlite-dev \
    libressl-dev \
"

RUN set -ex ;\
    # find MAJOR and MINOR python versions based on $PYTHON_VERSION
    export PYTHON_MAJOR_VERSION=$(echo "${PYTHON_VERSION}" | rev | cut -d"." -f3-  | rev) ;\
    export PYTHON_MINOR_VERSION=$(echo "${PYTHON_VERSION}" | rev | cut -d"." -f2-  | rev) ;\
    # replacing default repositories with edge ones
    echo "http://dl-cdn.alpinelinux.org/alpine/v$ALPINE_VERSION/community" >> /etc/apk/repositories ;\
    echo "http://dl-cdn.alpinelinux.org/alpine/v$ALPINE_VERSION/main" >> /etc/apk/repositories ;\

    # Add the packages, with a CDN-breakage fallback if needed
    apk add --no-cache $PACKAGES || \
        (sed -i -e 's/dl-cdn/dl-4/g' /etc/apk/repositories && apk add --no-cache $PACKAGES) ;\
    # Add packages just for the python build process with a CDN-breakage fallback if needed
    apk add --no-cache --virtual .build-deps $PYTHON_BUILD_PACKAGES || \
        (sed -i -e 's/dl-cdn/dl-4/g' /etc/apk/repositories && apk add --no-cache --virtual .build-deps $PYTHON_BUILD_PACKAGES) ;\

    # turn back the clock -- so hacky!
    echo "http://dl-cdn.alpinelinux.org/alpine/v$ALPINE_VERSION/main/" > /etc/apk/repositories ;\
    # echo "@community http://dl-cdn.alpinelinux.org/alpine/v$ALPINE_VERSION/community" >> /etc/apk/repositories ;\
    # echo "@testing http://dl-cdn.alpinelinux.org/alpine/v$ALPINE_VERSION/testing" >> /etc/apk/repositories ;\
    # echo "@edge-main http://dl-cdn.alpinelinux.org/alpine/edge/main" >> /etc/apk/repositories ;\

    # use pyenv to download and compile specific python version
    git clone --depth 1 https://github.com/pyenv/pyenv /usr/lib/pyenv ;\
    PYENV_ROOT=/usr/lib/pyenv /usr/lib/pyenv/bin/pyenv install $PYTHON_VERSION ;\
    # move specific version to correct path delete pyenv, no longer needed
    mv /usr/lib/pyenv/versions/$PYTHON_VERSION/ $PYTHON_PATH ;\
    rm -rfv /usr/lib/pyenv ;\
    # change the path on the header of every file from PYENV_ROOT to PYTHON_PATH
    cd $PYTHON_PATH/bin/ && sed -i "s+/usr/lib/pyenv/versions/$PYTHON_VERSION/+$PYTHON_PATH/+g" * ;\
    # delete binary "duplicates" and replace them with symlinks
    # this also optimizes space since they are actually the same binary
    rm -f $PYTHON_PATH/bin/python$PYTHON_MAJOR_VERSION \
          $PYTHON_PATH/bin/python$PYTHON_MINOR_VERSION \
          $PYTHON_PATH/bin/python$PYTHON_MAJOR_VERSION-config \
          $PYTHON_PATH/bin/python$PYTHON_MINOR_VERSION-config ;\
    ln -sf $PYTHON_PATH/bin/python $PYTHON_PATH/bin/python$PYTHON_MAJOR_VERSION ;\
    ln -sf $PYTHON_PATH/bin/python $PYTHON_PATH/bin/python$PYTHON_MINOR_VERSION ;\
    ln -sf $PYTHON_PATH/bin/python-config $PYTHON_PATH/bin/python$PYTHON_MAJOR_VERSION-config ;\
    ln -sf $PYTHON_PATH/bin/python-config $PYTHON_PATH/bin/python$PYTHON_MINOR_VERSION-config ;\
    # delete files to to reduce container size
    # tips taken from main python docker repo
    find /usr/lib/python$PYTHON_VERSION -depth \( -name '*.pyo' -o -name '*.pyc' -o -name 'test' -o -name 'tests' \) -exec rm -rf '{}' + ;\

    # remove build dependencies and any leftover apk cache
    apk del --no-cache --purge .build-deps ;\
    rm -rf /var/cache/apk/*

# since we will be "always" mounting the volume, we can set this up
ENTRYPOINT ["/usr/bin/dumb-init"]
CMD ["python"]

PS: With this change I also removed the pip upgrade right off the bat. I found out whenever pip is updated there are problems with package dependencies. This is why python itself doesn't have the very latest pip version and keeps a more old/stable one. (For example, just now with pip 10 there is a problem with pandas)

[jfloff]

Just reporting that python3.6 has an increase of 20Mb due to compiled python3.6-m version.

@tprobinson For the slim versions I would need your help testing this change. But with the above Dockerfile should be relatively easy for you to get this done. Ping back when you have some container size numbers.

[charlesreid1]

This looks fantastic. I don't mean to be a bump on a log but the Dockerfile you included is a lot more detailed than I would have made. This is all 👍 . alpine-python has become my go-to container image for several projects and it would open some really interesting possibilities for some projects.

[tprobinson]

Hmm, I think I can see a couple things, but overall this is a big improvement! The slim version shouldn't have any issues so long as python and pip are on the $PATH, but I'll test this out, see if I can find anything else that can be trimmed and make sure the slim version works with this.

[jfloff]

Should I commit the changes I did for the standard and on-build tags, and then you build on that and also change the slim tags?

[jfloff]

I guess I will go ahead and commit these changes and then @tprobinson can update the slim tags as soon as he can - no pressure!