Skip to content

Change to common user gid and uid for dy-static-file-server* & nginx replacement #140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Sep 27, 2021
2 changes: 1 addition & 1 deletion services/dy-static-file-server/.cookiecutterrc
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,4 @@ default_context:
project_slug: 'dy-static-file-server'
project_type: 'computational'
release_date: '2021'
version: '1.0.7'
version: '2.0.0'
2 changes: 1 addition & 1 deletion services/dy-static-file-server/VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.0.7
2.0.0
12 changes: 6 additions & 6 deletions services/dy-static-file-server/docker-compose-meta.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@ services:
io.simcore.name: '{"name": "dy-static-file-server"}'
io.simcore.outputs: '{"outputs": {}}'
io.simcore.type: '{"type": "dynamic"}'
io.simcore.version: '{"version": "1.0.7"}'
io.simcore.version: '{"version": "2.0.0"}'
org.label-schema.build-date: ${BUILD_DATE}
org.label-schema.schema-version: '1.0'
org.label-schema.vcs-ref: ${VCS_REF}
org.label-schema.vcs-url: ${VCS_URL}
simcore.service.settings: '[{"name": "resources", "type": "Resources", "value":
{"mem_limit":17179869184, "cpu_limit": 1000000000}}, {"name": "ports", "type":
"int", "value": 80}, {"name": "constraints", "type": "string", "value":
"int", "value": 8080}, {"name": "constraints", "type": "string", "value":
["node.platform.os == linux"]}]'
dy-static-file-server-dynamic-sidecar:
build:
Expand Down Expand Up @@ -55,7 +55,7 @@ services:
{"displayOrder": 5, "label": "File output", "description": "File from input",
"type": "data:*/*", "fileToKeyMap": {"test_file": "file_output"}}}}'
io.simcore.type: '{"type": "dynamic"}'
io.simcore.version: '{"version": "1.0.7"}'
io.simcore.version: '{"version": "2.0.0"}'
org.label-schema.build-date: ${BUILD_DATE}
org.label-schema.schema-version: '1.0'
org.label-schema.vcs-ref: ${VCS_REF}
Expand All @@ -64,7 +64,7 @@ services:
"/www/inputs", "state_paths": ["/workdir/generated-data"]}'
simcore.service.settings: '[{"name": "resources", "type": "Resources", "value":
{"mem_limit":17179869184, "cpu_limit": 1000000000}}, {"name": "ports", "type":
"int", "value": 80}, {"name": "constraints", "type": "string", "value":
"int", "value": 8080}, {"name": "constraints", "type": "string", "value":
["node.platform.os == linux"]}]'
dy-static-file-server-dynamic-sidecar-compose-spec:
build:
Expand Down Expand Up @@ -100,7 +100,7 @@ services:
{"displayOrder": 5, "label": "File output", "description": "File from input",
"type": "data:*/*", "fileToKeyMap": {"test_file": "file_output"}}}}'
io.simcore.type: '{"type": "dynamic"}'
io.simcore.version: '{"version": "1.0.7"}'
io.simcore.version: '{"version": "2.0.0"}'
org.label-schema.build-date: ${BUILD_DATE}
org.label-schema.schema-version: '1.0'
org.label-schema.vcs-ref: ${VCS_REF}
Expand All @@ -111,6 +111,6 @@ services:
"/www/inputs", "state_paths": ["/workdir/generated-data"]}'
simcore.service.settings: '[{"name": "resources", "type": "Resources", "value":
{"mem_limit":17179869184, "cpu_limit": 1000000000}}, {"name": "ports", "type":
"int", "value": 80}, {"name": "constraints", "type": "string", "value":
"int", "value": 8080}, {"name": "constraints", "type": "string", "value":
["node.platform.os == linux"]}]'
version: '3.7'
53 changes: 35 additions & 18 deletions services/dy-static-file-server/docker/custom/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# TODO: Please set your custom image here and adapt the Dockerfile/entrypoint.sh accordingly
FROM nginx:1.21.0-alpine as production
FROM joseluisq/static-web-server:2.0.2-alpine as production
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: would that not mean a new Major version? or at least minor?

Copy link
Contributor Author

@GitHK GitHK Sep 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After thinking about it more, I'd go for a major change here! going for 2.0.0
Base image and a bit of how everything behaves has been changed. Also trying to figure out compatibility with director-v0

#
# USAGE:
# cd services/dy-static-file-server
Expand All @@ -8,38 +8,55 @@ FROM nginx:1.21.0-alpine as production
#

ARG PYTHON_VERSION="3.8.10-r0"
ARG INPUT_DIR="/www/inputs"
ARG OUTPUT_DIR="/www/outputs"
ARG WORKDIR="/workdir"

ENV SC_BUILD_TARGET=production
ENV SERVER_ROOT="/www"
ENV INPUT_FOLDER="${SERVER_ROOT}/inputs"
ENV OUTPUT_FOLDER="${SERVER_ROOT}/outputs"
ENV SC_USER_ID=101
ENV SERVER_PORT=8080
ENV SERVER_LOG_LEVEL=debug

# creating own project's user
ENV SC_USER_ID 9004
ENV SC_USER_NAME scudy
RUN adduser -D -u ${SC_USER_ID} -s /bin/sh -h /home/${SC_USER_NAME} ${SC_USER_NAME}


LABEL maintainer=GitHK
COPY docker/custom/nginx.conf /etc/nginx/nginx.conf
COPY docker/custom/boot.sh /boot.sh

RUN apk add --update --no-cache \
"python3=${PYTHON_VERSION}" \
py3-pip
py3-pip \
su-exec
RUN pip3 install --upgrade \
pip \
virtualenv

RUN mkdir -p /venv
RUN python3 -m venv /venv

# add python app requirements
COPY requirements/base.txt /tmp/requirements.txt
RUN /venv/bin/pip3 install -r /tmp/requirements.txt
# create and activate virtual environment
RUN mkdir -p /venv && \
python3 -m venv /opt/venv
ENV PATH="/venv/bin:$PATH"

COPY static-content/hello-world.txt /www/hello-world.txt

RUN mkdir -p ${INPUT_DIR} && \
mkdir -p ${OUTPUT_DIR} && \
mkdir -p ${WORKDIR}
# add additional directories
RUN mkdir -p ${WORKDIR} && chown ${SC_USER_NAME}:${SC_USER_NAME} ${WORKDIR} && \
mkdir -p /docker && chown ${SC_USER_NAME}:${SC_USER_NAME} /docker

COPY --chown=${SC_USER_NAME}:${SC_USER_NAME} docker/custom/*.sh /docker

# add python app requirements
COPY requirements/base.txt /tmp/requirements.txt
RUN pip3 install -r /tmp/requirements.txt

COPY src/dy_static_file_server ${WORKDIR}/dy_static_file_server
COPY --chown=${SC_USER_NAME}:${SC_USER_NAME} static-content/hello-world.txt /www/hello-world.txt
COPY --chown=${SC_USER_NAME}:${SC_USER_NAME} src/dy_static_file_server ${WORKDIR}/dy_static_file_server

ENV NGINX_SERVER_ROOT="/www"

WORKDIR ${WORKDIR}/dy_static_file_server

CMD [ "/bin/sh", "/boot.sh" ]
EXPOSE 8080
ENTRYPOINT ["/bin/sh", "/docker/entrypoint.sh"]
CMD ["/docker/boot.sh"]
28 changes: 10 additions & 18 deletions services/dy-static-file-server/docker/custom/boot.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -8,25 +8,17 @@ echo User : "$(id "$(whoami)")"
echo Workdir : "$(pwd)"
echo Env : "$(env)"

echo "/workdir content"
ls -lah

if [ -n "${SIMCORE_NODE_BASEPATH+set}" ]
then
echo
echo moving website to "${NGINX_SERVER_ROOT}${SIMCORE_NODE_BASEPATH}"...
echo
echo "ensure some random data is created in /workdir/generated-data content"
python3 ensure_random_workdir_data.py

mkdir -p "${NGINX_SERVER_ROOT}${SIMCORE_NODE_BASEPATH}"
mv "${NGINX_SERVER_ROOT}"/*.txt "${NGINX_SERVER_ROOT}${SIMCORE_NODE_BASEPATH}"
echo "/workdir/generated-data content"
ls -lah /workdir/generated-data/

echo "Nginx will serve from ${NGINX_SERVER_ROOT}${SIMCORE_NODE_BASEPATH}"

else
echo "Nginx will serve from ${NGINX_SERVER_ROOT}, nothing to do"
fi
echo "starting background inputs->ouputs mapping when inputs change"
python3 inputs_to_outputs.py &

# ensure some random data is created in /workdir
/venv/bin/python ensure_random_workdir_data.py

# keep mirroring running in the background
exec /venv/bin/python folder_mirror.py &
exec nginx -g "daemon off;"
echo "booting static-web-server"
exec static-web-server
106 changes: 106 additions & 0 deletions services/dy-static-file-server/docker/custom/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
#!/bin/sh
set -o errexit
set -o nounset

IFS=$(printf '\n\t')
# This entrypoint script:
#
# - Executes *inside* of the container upon start as --user [default root]
# - Notice that the container *starts* as --user [default root] but
# *runs* as non-root user [$SC_USER_NAME]
#
echo Entrypoint for stage "${SC_BUILD_TARGET}" ...
echo User : "$(id "$(whoami)")"
echo Workdir : "$(pwd)"


# adapt to be compatible for legacy boot mode
if [ -n "$SIMCORE_NODE_BASEPATH" ]; then
echo "Boot mode: LEGACY"
echo "Creating ${INPUT_FOLDER} and ${OUTPUT_FOLDER}"
mkdir -p "${INPUT_FOLDER}"
mkdir -p "${OUTPUT_FOLDER}"
echo "SERVER_ROOT: ${SERVER_ROOT}"
echo "SIMCORE_NODE_BASEPATH: ${SIMCORE_NODE_BASEPATH}"
echo "Creating ${SERVER_ROOT}${SIMCORE_NODE_BASEPATH} and changin ownership to $SC_USER_NAME"
mkdir -p "${SERVER_ROOT}${SIMCORE_NODE_BASEPATH}"
chown -R "$SC_USER_NAME" "${SERVER_ROOT}${SIMCORE_NODE_BASEPATH}"
else
echo "Boot mode: DYNAMIC-SIDECAR"
fi


# expect input/output folders to be mounted
#TODO: determine if legacy boot more and based on that do stuff like creating
stat "${INPUT_FOLDER}" > /dev/null 2>&1 || \
(echo "ERROR: You must mount '${INPUT_FOLDER}' to deduce user and group ids" && exit 1)
stat "${OUTPUT_FOLDER}" > /dev/null 2>&1 || \
(echo "ERROR: You must mount '${OUTPUT_FOLDER}' to deduce user and group ids" && exit 1)

# NOTE: expects docker run ... -v /path/to/input/folder:${INPUT_FOLDER}
# check input/output folders are owned by the same user
if [ "$(stat -c %u "${INPUT_FOLDER}")" -ne "$(stat -c %u "${OUTPUT_FOLDER}")" ]
then
echo "ERROR: '${INPUT_FOLDER}' and '${OUTPUT_FOLDER}' have different user id's. not allowed" && exit 1
fi
# check input/outputfolders are owned by the same group
if [ "$(stat -c %g "${INPUT_FOLDER}")" -ne "$(stat -c %g "${OUTPUT_FOLDER}")" ]
then
echo "ERROR: '${INPUT_FOLDER}' and '${OUTPUT_FOLDER}' have different group id's. not allowed" && exit 1
fi

echo "listing inputs folder"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MINOR: not important in here but just FYI, in osparc-simcore images we use always prefix for the echo's so they can be easily parse by graylog
SEE https://github.com/ITISFoundation/osparc-simcore/blob/master/services/web/server/docker/entrypoint.sh#L7

ls -lah "${INPUT_FOLDER}"
echo "listing outputs folder"
ls -lah "${OUTPUT_FOLDER}"

echo "setting correct user id/group id..."
HOST_USERID=$(stat -c %u "${INPUT_FOLDER}")
HOST_GROUPID=$(stat -c %g "${INPUT_FOLDER}")
CONT_GROUPNAME=$(getent group "${HOST_GROUPID}" | cut -d: -f1)
if [ "$HOST_USERID" -eq 0 ]
then
echo "Warning: Folder mounted owned by root user... adding $SC_USER_NAME to root..."
addgroup "$SC_USER_NAME" root
else
echo "Folder mounted owned by user $HOST_USERID:$HOST_GROUPID-'$CONT_GROUPNAME'..."
# take host's credentials in $SC_USER_NAME
if [ -z "$CONT_GROUPNAME" ]
then
echo "Creating new group my$SC_USER_NAME"
CONT_GROUPNAME=my$SC_USER_NAME
addgroup -g "$HOST_GROUPID" "$CONT_GROUPNAME"
else
echo "group already exists"
fi

echo "changing $SC_USER_NAME $SC_USER_ID:$SC_USER_ID to $HOST_USERID:$HOST_GROUPID"
# in alpine there is no such thing as usermod... so we delete the user and re-create it as part of $CONT_GROUPNAME
deluser "$SC_USER_NAME" > /dev/null 2>&1
adduser -u "$HOST_USERID" -G "$CONT_GROUPNAME" -D -s /bin/sh "$SC_USER_NAME"

echo "Changing group properties of files around from $SC_USER_ID to group $CONT_GROUPNAME"
find / -path /var/log/nginx -prune -o -group "$SC_USER_ID" -print
find / -path /var/log/nginx -prune -o -group "$SC_USER_ID" -exec chgrp -h "$CONT_GROUPNAME" {} \;
# change user property of files already around
echo "Changing ownership properties of files around from $SC_USER_ID to group $CONT_GROUPNAME"
find / -path /var/log/nginx -prune -o -user "$SC_USER_ID" -exec chown -h "$SC_USER_NAME" {} \;
find / -path /var/log/nginx -prune -o -user "$SC_USER_ID" -print
fi

echo "Starting $* ..."
echo " $SC_USER_NAME rights : $(id "$SC_USER_NAME")"
echo " local dir : $(ls -al)"
echo " input dir : $(ls -al "${INPUT_FOLDER}")"
echo " output dir : $(ls -al "${OUTPUT_FOLDER}")"


# from original etrypoint
set -e

# Check if incomming command contains flags.
if [ "${1#-}" != "$1" ]; then
set -- static-web-server "$@"
fi

su-exec "$SC_USER_NAME" "$@"
24 changes: 0 additions & 24 deletions services/dy-static-file-server/docker/custom/nginx.conf

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: dy-static-file-server-dynamic-sidecar-compose-spec
key: simcore/services/dynamic/dy-static-file-server-dynamic-sidecar-compose-spec
type: dynamic
integration-version: 1.0.0
version: 1.0.7
version: 2.0.0
description: Modern test dynamic service providing a docker-compose specification file (with dynamic sidecar and compose-spec). Changes to the inputs will be forwarded to the outputs. The /workdir/generated-data directory is populated if no content is present.
contact: [email protected]
authors:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: dy-static-file-server-dynamic-sidecar
key: simcore/services/dynamic/dy-static-file-server-dynamic-sidecar
type: dynamic
integration-version: 1.0.0
version: 1.0.7
version: 2.0.0
description: Modern test dynamic service (with dynamic sidecar). Changes to the inputs will be forwarded to the outputs. The /workdir/generated-data directory is populated if no content is present.
contact: [email protected]
authors:
Expand Down
2 changes: 1 addition & 1 deletion services/dy-static-file-server/metadata/metadata.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: dy-static-file-server
key: simcore/services/dynamic/dy-static-file-server
type: dynamic
integration-version: 1.0.0
version: 1.0.7
version: 2.0.0
description: Legacy test dynamic service (starts using original director-v0). The /workdir/generated-data directory is populated if no content is present.
contact: [email protected]
authors:
Expand Down
Loading