Create setup for sidecar

[pcrespov]

The origin of this issue is that the sidecar could import simcore-sdk but not its dependencies because they were actually not installed. This issue blocks #186. In order to solve it properly we need to fulfill:

• Missing setup.py
• Missing requirement-dev.txt
• Non-root user in docker.
  • devel target will not create eggs as root and in production :

usr/local/lib/python3.6/site-packages/celery/platforms.py:795: RuntimeWarning: You're running the worker with superuser privileges: this is
sidecar_1    | absolutely not recommended!

• venv in docker wont' do. For this case, I agree that docker provides desired isolation
• rearrange package structure: main.py, tasks.py, celery.py and utils.py

[pcrespov]

Error running boot.sh.

>>> import sidecar
DEBUG:docker.utils.config:Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
DEBUG:docker.utils.config:No config file found
DEBUG:docker.utils.config:Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
DEBUG:docker.utils.config:No config file found
DEBUG:urllib3.connectionpool:http://localhost:None "GET /version HTTP/1.1" 200 543
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/work/services/sidecar/src/sidecar/__init__.py", line 1, in <module>
    from .core import *
  File "/work/services/sidecar/src/sidecar/core.py", line 455, in <module>
    SIDECAR = Sidecar()
  File "/work/services/sidecar/src/sidecar/core.py", line 43, in __init__
    self._s3 = S3Settings()
  File "/work/services/sidecar/src/sidecar/utils.py", line 70, in __init__
    access_key=self._config.access_key, secret_key=self._config.secret_key)
  File "/work/packages/s3wrapper/src/s3wrapper/s3_client.py", line 23, in __init__
    secure=secure)
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 135, in __init__
    is_valid_endpoint(endpoint)
  File "/usr/local/lib/python3.6/site-packages/minio/helpers.py", line 301, in is_valid_endpoint
    if hostname[-1] == '.':
IndexError: string index out of range

solved restructuring sidecar service (last point above)

[pcrespov]

Error

sidecar_1    | Traceback (most recent call last):
sidecar_1    |   File "/home/scu/.venv/lib/python3.6/site-packages/celery/app/trace.py", line 374, in trace_task
sidecar_1    |     R = retval = fun(*args, **kwargs)
sidecar_1    |   File "/home/scu/.venv/lib/python3.6/site-packages/celery/app/trace.py", line 629, in __protected_call__
sidecar_1    |     return self.run(*args, **kwargs)
sidecar_1    |   File "/home/scu/.venv/lib/python3.6/site-packages/sidecar/tasks.py", line 11, in pipeline
sidecar_1    |     from .core import SIDECAR
sidecar_1    |   File "/home/scu/.venv/lib/python3.6/site-packages/sidecar/core.py", line 451, in <module>
sidecar_1    |     SIDECAR = Sidecar()
sidecar_1    |   File "/home/scu/.venv/lib/python3.6/site-packages/sidecar/core.py", line 32, in __init__
sidecar_1    |     self._docker = DockerSettings()
sidecar_1    |   File "/home/scu/.venv/lib/python3.6/site-packages/sidecar/utils.py", line 59, in __init__
sidecar_1    |     self.client = docker.from_env(version='auto')
sidecar_1    |   File "/home/scu/.venv/lib/python3.6/site-packages/docker/client.py", line 81, in from_env
sidecar_1    |     **kwargs_from_env(**kwargs))
sidecar_1    |   File "/home/scu/.venv/lib/python3.6/site-packages/docker/client.py", line 38, in __init__
sidecar_1    |     self.api = APIClient(*args, **kwargs)
sidecar_1    |   File "/home/scu/.venv/lib/python3.6/site-packages/docker/api/client.py", line 154, in __init__
sidecar_1    |     self._version = self._retrieve_server_version()
sidecar_1    |   File "/home/scu/.venv/lib/python3.6/site-packages/docker/api/client.py", line 179, in _retrieve_server_version
sidecar_1    |     'Error while fetching server API version: {0}'.format(e)
sidecar_1    | docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

checks

• scu in docker group

crespo@crespo-ubuntu:~/devp/osparc-simcore$ docker logs 027
INFO: Sidecar running as uid=8004(scu) gid=8004(scu) groups=8004(scu),1001(docker)
INFO: Booting in production mode ...
[2018-08-29 16:55:29,003: ERROR/MainProcess] consumer: Cannot connect to amqp://simcore:**@rabbit:5672//: [Errno 111] Connection refused.
Trying again in 2.00 seconds...

[2018-08-29 16:55:31,015: ERROR/MainProcess] consumer: Cannot connect to amqp://simcore:**@rabbit:5672//: [Errno 111] Connection refused.
Trying again in 4.00 seconds...

[2018-08-29 16:55:35,040: INFO/MainProcess] Connected to amqp://simcore:**@rabbit:5672//
[2018-08-29 16:55:35,049: INFO/MainProcess] mingle: searching for neighbors
[2018-08-29 16:55:36,075: INFO/MainProcess] mingle: all alone
[2018-08-29 16:55:36,115: INFO/MainProcess] celery@027f68d802f7 ready.

• isolated example works

Solution

Entrypoint with su-exec scu "$@" instead of su-exec scu:scu "$@" ... obviously! :(

[pcrespov]

Error

sidecar_1 | ImportError: Error loading shared library libpq.so.5: No such file or directory (needed by /home/scu/.venv/lib/python3.6/site-packages/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so)

It actually exists but does not work since there are some dlls that are not present. The problem is in the production stage. Solved in commit bab7831

[oetiker]

this sounds is if the postgresql library was missing

[oetiker]

which might make sense since the sidecar should not access postgres directly

[mguidon]

It is recommended to build psycopg2 against the C low level library. You do that in your build stage by installing them into alpine. In your run stage you need the corresponding runtime libraries.
@oetiker : What is the disadvantage of letting the sidecar talk to the db?

[sanderegg]

make sidecar run under Windows: detect windows, and run as root in entrypoint.sh