Skip to content

BLD: Refresh gitpod build files #56375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 11 commits into from
Closed
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .gitpod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,17 @@
# images for gitpod pandas are in https://hub.docker.com/r/pandas/pandas-gitpod/tags
# we're using the Dockerfile in the base of the repo
image:
file: Dockerfile
file: gitpod/gitpod.Dockerfile
tasks:
- name: Prepare development environment
init: |
mkdir -p .vscode
cp gitpod/settings.json .vscode/settings.json
git fetch --tags
conda init
source ~/.bashrc
conda env update --file environment.yml --prune
conda activate pandas-dev
python -m pip install -ve . --no-build-isolation --config-settings editable-verbose=true
pre-commit install
command: |
Expand Down
69 changes: 28 additions & 41 deletions gitpod/gitpod.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,48 +1,35 @@
# Doing a local shallow clone - keeps the container secure
# and much slimmer than using COPY directly or making a
# remote clone
ARG BASE_CONTAINER="pandas/pandas-dev:latest"
FROM gitpod/workspace-base:latest as clone

# the clone should be deep enough for versioneer to work
RUN git clone https://github.com/pandas-dev/pandas --depth 12 /tmp/pandas

# -----------------------------------------------------------------------------
# Using the pandas-dev Docker image as a base
# This way, we ensure we have all the needed compilers and dependencies
# while reducing the build time
FROM ${BASE_CONTAINER} as build

# -----------------------------------------------------------------------------
USER root

# -----------------------------------------------------------------------------
# ---- ENV variables ----
# ---- Directories needed ----
ENV WORKSPACE=/workspace/pandas/ \
CONDA_ENV=pandas-dev

# Allows this micromamba.Dockerfile to activate conda environments
SHELL ["/bin/bash", "--login", "-o", "pipefail", "-c"]

# Copy over the shallow clone
COPY --from=clone --chown=gitpod /tmp/pandas ${WORKSPACE}

# Everything happens in the /workspace/pandas directory
WORKDIR ${WORKSPACE}

# Build pandas to populate the cache used by ccache
RUN git config --global --add safe.directory /workspace/pandas
RUN conda activate ${CONDA_ENV} && \
python -m pip install -e . --no-build-isolation && \
python setup.py build_ext --inplace && \
ccache -s

# Gitpod will load the repository into /workspace/pandas. We remove the
# directory from the image to prevent conflicts
RUN rm -rf ${WORKSPACE}

# -----------------------------------------------------------------------------
# Always return to non privileged user
RUN chown -R gitpod:gitpod /home/gitpod/.cache/
# Install base utilities
RUN apt-get update \
&& apt-get install -y build-essential \
&& apt-get install -y wget \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

# Install miniconda
ENV CONDA_DIR /opt/conda
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not opposed but whats the advantage of running conda within Docker versus just using docker for the virtualization?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess to install new packages?

For some reason I can't do stuff like apt install in gitpod.
(sudo doesnt work for me)

In general, I think it's preferred to have stuff installed from conda over pip, too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does apt-get install work?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also on the note of conda versus pip - conda is a pretty big install. Most images try to be small and easy to deploy; I would imagine this makes the image significantly larger than the base

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are users expected to modify this environment? If so I think installing conda would be nice for ease of use. But since this is for development anyways and not a service having conda here would be OK to me

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does apt-get install work?

I don't think so?
It's been a while since I last used gitpod. Sorry for the late reply.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are users expected to modify this environment? If so I think installing conda would be nice for ease of use. But since this is for development anyways and not a service having conda here would be OK to me

I tend to want to install new packages at least
(e.g. things like a profiler, or trying out different versions of dependencies, like numpydev, etc.)

RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda

# Put conda in path so we can use conda activate
ENV PATH=$CONDA_DIR/bin:$PATH

# init conda
RUN conda init
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you combine these RUN statements to reduce the image layers?


# Use the speedy libmamba solver
RUN conda install -n base conda-libmamba-solver
RUN conda config --set solver libmamba

# # -----------------------------------------------------------------------------
# # Always return to non privileged user
RUN chown -R gitpod:gitpod /opt/conda/pkgs
RUN chown -R gitpod:gitpod /opt/conda/envs
RUN chown -R gitpod:gitpod /home/gitpod/.conda
RUN chown -R gitpod:gitpod /home/gitpod/.cache
USER gitpod