[Feature]: Host CPU Docker image on Docker Hub

[VMinB12]

🚀 The feature, motivation and pitch

Currently vllm has pre-built docker images for vllm/vllm-openai. This image requires a GPU to run. Now that CPU support is being added, it would be great if CPU images could be added too.
Allowing users to pull CPU images could greatly simplify the process for users and relaxing the GPU requirement would allow more users to benefit.

Alternatives

No response

Additional context

No response

[VMinB12]

Any updates on this issue? I really believe the community would greatly benefit from a prebuilt and pullable cpu docker image.

[KenMacD]

I wouldn't limit this to cpu only though. It'd be nice if all the docker images were built and included as part of a release. If not published to dockerhub that could at least be hosted on github.

[cquil11]

Are there any updates on this?

[rpvelloso]

+1

[medihack]

We are currently serving vLLM as container in production, but llama.cpp with CPU support and a small model in development because of this issue. It would be great to have an image available for CPU (and maybe other platforms), too.

[medihack]

With this fixed and two releases since then (v0.6.6 and v0.6.6.post1), I wonder why there is no CPU image yet on Docker Hub. When will it be released there?

[VMinB12]

I see the issue has been closed as completed, but I cannot find the CPU image on docker hub. I see the #11261 PR mentions a manual process is needed to push images to docker hub. @DarkLight1337 Would you be so kind as to trigger this manual process?

[DarkLight1337]

@simon-mo can you help?

[simon-mo]

It looks like we are not pushing to docker hub yet? Cc @khluu

[taha-yassine]

I think it's being worked on #12615 (comment)

[VMinB12]

@simon-mo @khluu Any updates on this? At the time of writing I do not see cpu images on https://hub.docker.com/u/vllm
Thanks for working on this, I would much appreciate a CPU docker image to be provided.

[rpvelloso]

+1

[VMinB12]

@simon-mo @khluu Sorry for the insistence, but I still do not see a CPU Docker image on the docker hub. Any updates would be much appreciated.

[hugodendievelb12]

As a workaround, cloning the repo inside a Dockerfile works:

######################### BASE IMAGE #########################
FROM ubuntu:22.04 AS base

WORKDIR /workspace/

ARG PYTHON_VERSION=3.12
ARG PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu"

ENV LD_PRELOAD=""

# Install minimal dependencies and uv
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update -y \
    && apt-get install -y --no-install-recommends ccache git curl wget ca-certificates \
        gcc-12 g++-12 libtcmalloc-minimal4 libnuma-dev ffmpeg libsm6 libxext6 libgl1 \
    && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12 \
    && curl -LsSf https://astral.sh/uv/install.sh | sh

RUN git clone --branch v0.8.5.post1 https://github.com/vllm-project/vllm.git /workspace/vllm

ENV CCACHE_DIR=/root/.cache/ccache
ENV CMAKE_CXX_COMPILER_LAUNCHER=ccache

ENV PATH="/root/.local/bin:$PATH"
ENV VIRTUAL_ENV="/opt/venv"
ENV UV_PYTHON_INSTALL_DIR=/opt/uv/python
RUN uv venv --python ${PYTHON_VERSION} --seed ${VIRTUAL_ENV}
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

ENV UV_HTTP_TIMEOUT=500

# Install Python dependencies 
ENV PIP_EXTRA_INDEX_URL=${PIP_EXTRA_INDEX_URL}
ENV UV_EXTRA_INDEX_URL=${PIP_EXTRA_INDEX_URL}
ENV UV_INDEX_STRATEGY="unsafe-best-match"
ENV UV_LINK_MODE="copy"
RUN --mount=type=cache,target=/root/.cache/uv \
    uv pip install --upgrade pip && \
    uv pip install -r /workspace/vllm/requirements/cpu.txt

RUN --mount=type=cache,target=/root/.cache/uv \
    uv pip install intel-openmp==2024.2.1 intel_extension_for_pytorch==2.6.0

ENV LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4:/opt/venv/lib/libiomp5.so:$LD_PRELOAD"

RUN echo 'ulimit -c 0' >> ~/.bashrc

######################### BUILD IMAGE #########################
FROM base AS vllm-build

ARG GIT_REPO_CHECK=0
# Support for building with non-AVX512 vLLM: docker build --build-arg VLLM_CPU_DISABLE_AVX512="true" ...
ARG VLLM_CPU_DISABLE_AVX512
ENV VLLM_CPU_DISABLE_AVX512=${VLLM_CPU_DISABLE_AVX512}

WORKDIR /workspace/vllm

RUN --mount=type=cache,target=/root/.cache/uv \
    uv pip install -r /workspace/vllm/requirements/build.txt

RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=cache,target=/root/.cache/ccache \
    VLLM_TARGET_DEVICE=cpu python3 setup.py bdist_wheel 


######################### RELEASE IMAGE #########################
FROM base AS vllm-openai

WORKDIR /workspace/

RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=cache,target=/root/.cache/ccache \
    --mount=type=bind,from=vllm-build,src=/workspace/vllm/dist,target=dist \
    uv pip install dist/*.whl

ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"]