Squashed commit of PR ITISFoundation#199 with the following:

commit 0df033b
Author: Pedro Crespo <[email protected]>
Date:   Thu Aug 30 14:21:44 2018 +0200

    Review feedback: removed unnecessary config sections

commit bab7831
Author: Pedro Crespo <[email protected]>
Date:   Thu Aug 30 13:59:07 2018 +0200

    Fixes missing dlls

commit c0bc7d6
Author: Pedro Crespo <[email protected]>
Date:   Thu Aug 30 11:53:41 2018 +0200

    Fixes access to docker socket:
     - cleanup Dockerfile
     - minor doc and logs

commit 5b6f4c7
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 17:47:13 2018 +0200

    Minor cleanup and doc

commit 43f7305
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:52:30 2018 +0200

    Minor change in web server

commit 7f07022
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:52:16 2018 +0200

    sidecar: starts as root (via entrypoint) but runs as scu (non-root)

commit 90359c4
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:51:32 2018 +0200

    Fixes sidecar: temporary fix to connection to other services

commit 7619671
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:44:42 2018 +0200

    Fixes host in server running inside a docker

commit bdbf078
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:43:56 2018 +0200

    Separating maintenance services

commit 83e492d
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 13:46:55 2018 +0200

    Fixes on web server:
     - removed usused cs_s4l from config
     - added env for rabbit

commit 8fee402
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 11:48:46 2018 +0200

    Fixes on webserver: - Tmp  solution for production target based on production-build stage -  Normalized naming of package and console script

commit a5d8220
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 22:06:59 2018 +0200

    Implements entry in ITISFoundation#186: - upgrades Dockerfile with .venv and pip installing 3rd parties

commit 3c55bf3
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 18:46:57 2018 +0200

    Setting up boot for web service and forgot configs

commit 8003744
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 18:40:12 2018 +0200

    Adding config data and retreiving it as resource

commit 94de3d7
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 10:58:02 2018 +0200

    Minor changes in docker after discussion with MaG

commit e663a1e
Author: Pedro Crespo <[email protected]>
Date:   Mon Aug 27 22:08:38 2018 +0200

    Updated ownership

commit bf486b3
Author: Pedro Crespo <[email protected]>
Date:   Mon Aug 27 21:53:12 2018 +0200

    Fixes linting errors and added notes

commit 752c7bf
Author: Pedro Crespo <[email protected]>
Date:   Mon Aug 27 19:49:39 2018 +0200

    WIP ITISFoundation#198: non-root user
     - all volumes bound to /home/scu
     - All modules pip installed (in dev w/ edit mode).
     - All files produce in dev mode on bound volumes are not deletable
     - Production stage is further optimized by taking only venv from base
     - Fixes sidecar access to input/output/log volumes

commit ec755bc
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 21:53:13 2018 +0200

    WIP ITISFoundation#198:
     - added a boot.sh script to start celery worker
     - restructured package submodules. Otherwise got error (see comments in ITISFoundation#198)
     -

commit 9bbfbde
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 21:52:12 2018 +0200

    Minor update

commit 53c7707
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 20:29:26 2018 +0200

    Fixes import failure of trafaret and updages vs sqalquemy

commit a27a279
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 20:24:49 2018 +0200

    WIP ITISFoundation#198: setup for sidecar
     - created setup and requirements for dev&prod
     - installs package in docker (instead of reference via sys.path). this way all third-parties are installed.
     - minor cleanup of code

commit bbc7cb6
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 20:22:31 2018 +0200

    Minor: sync req versions

Author: Pedro Crespo

.env-devel

@@ -1,3 +1,7 @@
+# NOTE: write here host gid and docker gid.
+HOST_GID=1000
+DOCKER_GID=1001
+#--------
 POSTGRES_ENDPOINT=postgres:5432
 POSTGRES_USER=simcore
 POSTGRES_PASSWORD=simcore

.github/CODEOWNERS

@@ -2,11 +2,12 @@
 
 # files and folders recursively
 /docs/                   @pcrespov
-/services/web/client     @odeimaiz
-/services/web/server     @pcrespov
-/services/modeling       @sanderegg
 /services/authentication @pcrespov
 /services/computation    @mguidon
+/services/dy*            @sanderegg
+/services/sidecar        @pcrespov, @mguidon
+/services/web/client     @odeimaiz
+/services/web/server     @pcrespov
 
 # any change in travis
 /.travis.yml     @odeimaiz

packages/simcore-sdk/setup.py

@@ -1,10 +1,13 @@
-from setuptools import setup
+from setuptools import (
+    setup,
+    find_packages
+)
 
 INSTALL_REQUIRES = [
     'networkx==2.1',
     'psycopg2==2.7.4',
-    'sqlalchemy==1.2.8',
-    'tenacity==4.12.0'
+    'sqlalchemy==1.2.9',
+    'tenacity==4.12.0',
     'trafaret-config==2.0.1'
 ]
 
@@ -21,11 +24,13 @@
 setup(
     name='simcore-sdk',
     version='0.1.0',
-    package_dir={'': 'src'},
-    packages=['simcore_sdk'],
+    packages=find_packages(where='src'),
+    package_dir={
+        '': 'src',
+    },
     python_requires='>=3.6',
-    INSTALL_REQUIRES=INSTALL_REQUIRES,
-    TEST_REQUIRE=TEST_REQUIRE,
+    install_requires=INSTALL_REQUIRES,
+    tests_require=TEST_REQUIRE,
     extras_require= {
         'test': TEST_REQUIRE
     },

services/docker-compose.devel.yml

@@ -18,18 +18,22 @@ services:
   webserver:
     image: services_webserver:dev
     build:
+      args:
+        - HOST_GID_ARG=${HOST_GID:?Undefined host gid}
       target: development
     volumes:
-      - ./web/server:/home/scu/server
-      - ./web/client/source-output:/home/scu/client
+      - ./web/server:/home/scu/services/web/server
+      - ./web/client/source-output:/home/scu/services/web/client
       - ../packages:/home/scu/packages
     depends_on:
       - webclient
   #--------------------------------------------------------------------
   sidecar:
     image: services_sidecar:dev
     build:
+      args:
+        - HOST_GID_ARG=${HOST_GID:?Undefined host gid}
       target: development
     volumes:
-      - ./sidecar/src/sidecar:/work/sidecar
-      - ../packages:/work/packages
+      - ./sidecar:/home/scu/services/sidecar
+      - ../packages:/home/scu/packages

services/docker-compose.swarm.yml.template

@@ -87,9 +87,9 @@ services:
       - S3_SECRET_KEY=<enter secret key>
       - S3_BUCKET_NAME=simcore
     volumes:
-      - input:/input
-      - output:/output
-      - log:/log
+      - input:/home/scu/input
+      - output:/home/scu/output
+      - log:/home/scu/log
       - /var/run/docker.sock:/var/run/docker.sock
     ports:
       - "8000:8000"

services/docker-compose.tools.yml

@@ -0,0 +1,19 @@
+version: '3.4'
+services:
+  # Maintenance services
+  #--------------------------------------------------------------------
+  flower:
+    image: ondrejit/flower:latest
+    command: --broker=amqp://${RABBITMQ_USER}:${RABBITMQ_PASSWORD}@rabbit:5672
+    ports:
+      - 5555:5555
+    depends_on:
+      - rabbit
+  #--------------------------------------------------------------------
+  adminer:
+    image: adminer
+    ports:
+      - 18080:8080
+    depends_on:
+      - postgres
+  #--------------------------------------------------------------------

services/docker-compose.yml

@@ -36,22 +36,24 @@ services:
     build:
       context: ../
       dockerfile: services/web/Dockerfile
-      target: ci
+      target: production
     ports:
       - '9081:8080'
     environment:
       - DIRECTOR_HOST=director
       - DIRECTOR_PORT=8001
+      - POSTGRES_ENDPOINT=${POSTGRES_ENDPOINT}
+      - POSTGRES_USER=${POSTGRES_USER}
+      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+      - POSTGRES_DB=${POSTGRES_DB}
       - RABBITMQ_USER=${RABBITMQ_USER}
       - RABBITMQ_PASSWORD=${RABBITMQ_PASSWORD}
+      - RABBITMQ_PROGRESS_CHANNEL=${RABBITMQ_PROGRESS_CHANNEL}
+      - RABBITMQ_LOG_CHANNEL=${RABBITMQ_LOG_CHANNEL}
       - S3_ENDPOINT=${S3_ENDPOINT}
       - S3_ACCESS_KEY=${S3_ACCESS_KEY}
       - S3_SECRET_KEY=${S3_SECRET_KEY}
       - S3_BUCKET_NAME=${S3_BUCKET_NAME}
-      - POSTGRES_ENDPOINT=${POSTGRES_ENDPOINT}
-      - POSTGRES_USER=${POSTGRES_USER}
-      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
-      - POSTGRES_DB=${POSTGRES_DB}
     depends_on:
       - webclient
   #--------------------------------------------------------------------
@@ -63,14 +65,6 @@ services:
     ports:
       - "15672:15672"
   #--------------------------------------------------------------------
-#  flower:
-#    image: ondrejit/flower:latest
-#    command: --broker=amqp://${RABBITMQ_USER}:${RABBITMQ_PASSWORD}@rabbit:5672
-#    ports:
-#      - 5555:5555
-#    depends_on:
-#      - rabbit
-  #--------------------------------------------------------------------
   postgres:
     image: postgres:10
     environment:
@@ -82,29 +76,23 @@ services:
     ports:
       - "5432:5432"
   #--------------------------------------------------------------------
-  adminer:
-    image: adminer
-    ports:
-      - 18080:8080
-    depends_on:
-      - postgres
-  #--------------------------------------------------------------------
   sidecar:
     build:
       # the context for the build is the git repo root directory, this allows to copy
       # the packages directory into any docker image
       context: ../
       dockerfile: services/sidecar/Dockerfile
+      args:
+        - DOCKER_GID_ARG=${DOCKER_GID:?Undefined docker gid in host}
       target: production
     volumes:
-      - input:/input
-      - output:/output
-      - log:/log
+      - input:/home/scu/input
+      - output:/home/scu/output
+      - log:/home/scu/log
       - /var/run/docker.sock:/var/run/docker.sock
     ports:
       - "8000:8000"
     environment:
-      - PYTHONPATH=/work/packages/simcore-sdk/src:/work/packages/s3wrapper/src
       - RABBITMQ_USER=${RABBITMQ_USER}
       - RABBITMQ_PASSWORD=${RABBITMQ_PASSWORD}
       - POSTGRES_ENDPOINT=${POSTGRES_ENDPOINT}

services/dy-2Dgraph/use-cases/kember/requirements.txt

@@ -5,5 +5,5 @@ networkx==2.1
 pandas==0.22.0
 plotly==2.6.0
 psycopg2-binary==2.7.4
-sqlalchemy==1.2.8
-tenacity==4.12.0
+sqlalchemy==1.2.9
+tenacity==4.12.0

services/dy-3dvis/simcoreparaviewweb/requirements.txt

@@ -1,6 +1,6 @@
 minio==4.0.0
 networkx==2.1
 psycopg2-binary==2.7.4
-sqlalchemy==1.2.8
+sqlalchemy==1.2.9
 tenacity==4.12.0
-docker==3.3.0
+docker==3.3.0

services/sidecar/.docker/entrypoint.sh

@@ -0,0 +1,19 @@
+#!/bin/sh
+
+# This entrypoint script:
+#
+# - Executes with root privileges *inside* of the container upon start
+# - Allows starting the container as root to perform some root-level operations at runtime
+#  (e.g. on volumes mapped inside)
+# - Notice that this way, the container *starts* as root but *runs* as scu (non-root user)
+#
+# See https://stackoverflow.com/questions/39397548/how-to-give-non-root-user-in-docker-container-access-to-a-volume-mounted-on-the
+
+
+addgroup scu docker
+
+chown -R scu:scu /home/scu/input
+chown -R scu:scu /home/scu/output
+chown -R scu:scu /home/scu/log
+
+su-exec scu "$@"

services/sidecar/Dockerfile

@@ -1,33 +1,103 @@
-FROM python:3.6-alpine as common
+FROM python:3.6-alpine as base
 
-LABEL maintainer="Manuel Guidon <[email protected]"
+LABEL maintainer=mguidon
 
-WORKDIR /work/sidecar
+ARG DOCKER_GID_ARG=1001
 
-RUN apk add --no-cache postgresql-dev gcc libc-dev
+RUN apk add --no-cache \
+      su-exec
 
-COPY services/sidecar/requirements.txt requirements.txt
+# create user `scu` and `docker` group (with same id as in host)
+RUN adduser -D -u 8004 scu &&\
+    addgroup -g $DOCKER_GID_ARG docker
 
-RUN pip install --upgrade pip \
-    && pip install -r requirements.txt \
-    && pip list --format=columns
+ENV HOME /home/scu
+ENV PIP  /home/scu/.venv/bin/pip3
 
 EXPOSE 8000
+VOLUME /home/scu/input
+VOLUME /home/scu/output
+VOLUME /home/scu/log
 
-FROM common as development
+WORKDIR /home/scu
 
-VOLUME /work/sidecar
-VOLUME /work/packages
+# -------------------------- Build stage -------------------
+#
+# - Preserves relative folder structure
+#
+# + /home/scu/              $HOME
+#    + services/sidecar
+#        ...
+#    + packages
+#        ...
+FROM base as build
 
-# NO clue why this does not work without explicitly specifying
-ENTRYPOINT celery -A sidecar worker -c 2 --loglevel=info
+RUN apk add --no-cache \
+      postgresql-dev \
+      gcc \
+      libc-dev
 
-FROM common as production
+RUN python3 -m venv $HOME/.venv &&\
+    $PIP install --no-cache-dir --upgrade \
+      pip \
+      wheel \
+      setuptools
 
-# the context for the build is the git repo root directory
-COPY services/sidecar/src /work
-COPY packages /work/packages
+# TODO: check if scu:scu copy is necessary!?
+COPY --chown=scu:scu services/sidecar/requirements/base.txt requirements-base.txt
+RUN $PIP install --no-cache-dir -r requirements-base.txt &&\
+    rm requirements-base.txt
 
-# NO clue why this does not work without explicitly specifying
-ENV PYTHONPATH="/work/packages/simcore-sdk/src:/work/packages/s3wrapper/src"
-ENTRYPOINT celery -A sidecar worker -c 2 --loglevel=info
+COPY --chown=scu:scu services/sidecar/.docker .docker
+COPY --chown=scu:scu services/sidecar/boot.sh boot.sh
+
+# --------------------------Development stage -------------------
+FROM build as development
+
+ARG HOST_GID_ARG=1000
+
+# in dev-mode we give access to `scu` to host's mapped volumes
+# FIXME: files created by scu cannot be deleted by host! we need to do the same group in host?
+RUN addgroup -g $HOST_GID_ARG hgrp &&\
+    addgroup scu hgrp && \
+    chown -R scu:scu $HOME/.venv
+
+VOLUME /home/scu/packages
+VOLUME /home/scu/services/sidecar
+
+ENV DEBUG 1
+USER root
+ENTRYPOINT [ "/bin/sh", ".docker/entrypoint.sh" ]
+CMD ./boot.sh
+
+
+# --------------------------Production multi-stage -------------------
+#FROM build as build-production
+FROM build as production
+
+# TODO: check if scu:scu copy is necessary in all cases!? since we are just installing?
+COPY --chown=scu:scu packages $HOME/packages
+COPY --chown=scu:scu services/sidecar $HOME/services/sidecar
+
+WORKDIR /home/scu/services/sidecar
+RUN $PIP --no-cache-dir install -r requirements/prod.txt ;\
+    $PIP list
+
+#-------------------
+#FROM base as production
+
+# TODO: PC Reduce docker size by installing only non-dev
+
+#COPY --from=build-production --chown=scu:scu $HOME/boot.sh boot.sh
+#COPY --from=build-production --chown=scu:scu $HOME/.venv .venv
+#COPY --from=build-production --chown=scu:scu $HOME/.docker .docker
+
+WORKDIR /home/scu/
+
+RUN . $HOME/.venv/bin/activate; pip list &&\
+    rm -rf $HOME/services
+
+ENV DEBUG 0
+USER root
+ENTRYPOINT [ "/bin/sh", ".docker/entrypoint.sh" ]
+CMD ./boot.sh

services/sidecar/README.md

@@ -1,3 +1,29 @@
 # Sidecar
 
 Use sidecar container to control computational service.
+
+TODO: See issue #198
+
+```bash
+
+# create an prepare a clean virtual environment ...
+python3 -m venv .venv
+source .venv/bin/activate
+pip3 install --upgrade pip setuptools wheel
+# ..or
+make .venv
+source .venv/bin/activate
+
+
+cd services/sidecar
+
+# for development (edit mode)
+# see how this packages is listed with a path to it src/ folder
+pip3 install -r requirements/dev.txt
+pip3 list
+
+
+# for production
+pip3 install -r requirements/prod.txt
+pip3 list
+```