♻️✨adding nodeports support to dynamic-sidecar

[GitHK]

What do these changes do?

Services booted via dynamic-sidecar will have the ports pulled and pushed for themselves. Also the state of the service is being handled by the dynamic-sidecar.

Service state integration

• When a service is started, before actually running docker-compose up and creating all the images, the dynamic-sidecar will pull the state of the service.
• When the service is closed, before running docker-compose down the state of the service is saved.
• The state of the service is stored in one or more directories, for example: see state_paths key here.
• Each entry inside state_paths will be mounted as a separate volume.

Nodeports integration

• When the service is available in the client interface (a.k.a. you can see the service's contents), the frontend will no longer request osparc.io/x/UUID/retrieve endpoint. Instead a call to osparc.io/projects/PROJECT_ID/nodes/NODE_ID/retrieve will be made. The webserver will forward the request to the director-v2 which in turn will call into they dynamic-sidecar's API. The event chain will pull data from input ports.
• When the service is closed, before running docker-compose down, output ports are pushed.
• The dynamic-sidecar adds a watcher on the outputs directory. When changes occur to this directory output ports are pushed.

Done/todos:

• refactor how services are stopped, also how cleanup is done when services fail
• improved boot time for dynamic-sidecar in production mode
• adds pulling and pushing via nodeports
• adds support for saving "state" directories
• changes retrieve entrypoint for dynamic-sidecar in fronted via browser
• services require to run under gid=8004 and uid=8004 like here
• dynamic-sidecar settings migrate to settingslib
• simcore-sdk's data_manager.pull now supports save_to option to save to a different folder
• add integration tests for nodeports pull/push
• add integration tests for status retrieval pull/push
• removed mocked API from dynamic-sidecar. No requests are directly landing on the dynamic-sidecar form the frontend.
• added exponential retreat when requesting retrieve and status from director-v2; these sometimes failed because the service was not immediately responsible.

Related issue/s

• requires ♻️ Maintenance/refactors servicelib #2516
• requires dy-static-file-server copies inputs to outputs & generates "workdir" data if not present osparc-services#138
• requires Change to common user gid and uid for dy-static-file-server* & nginx replacement osparc-services#140
• requires Fixes an issue with running dy-static-file-server on slower machines osparc-services#142
• requires 🐛Fixing nodeports_v2 port.set() race condition #2587
• requires ✨Replacing dynamic sidecar proxy with Caddy #2597
• closes Change how dynamic services run - dy-sidecar #244

How to test

Boot stack locally in development or production mode.

Use the following images. Note: you might need to manually assign permissions from the database:

• simcore/services/dynamic/dy-static-file-server-dynamic-sidecar:2.0.0
• simcore/services/dynamic/dy-static-file-server-dynamic-sidecar-compose-spec:2.0.0

Checklist

• Did you change any service's API? Then make sure to bundle document and upgrade version (make openapi-specs, git commit ... and then make version-*)
• Unit tests for the changes exist
• Runs in the swarm

[codecov]

Codecov Report

Merging #2509 (23e9b21) into master (5ef4cd9) will decrease coverage by 0.0%.
The diff coverage is 75.2%.

@@           Coverage Diff            @@
##           master   #2509     +/-   ##
========================================
- Coverage    77.0%   77.0%   -0.1%     
========================================
  Files         625     630      +5     
  Lines       24094   24741    +647     
  Branches     2362    2430     +68     
========================================
+ Hits        18567   19057    +490     
- Misses       4890    5040    +150     
- Partials      637     644      +7     

Flag	Coverage Δ
integrationtests	66.7% <69.4%> (+0.8%)	⬆️
unittests	71.4% <64.5%> (-0.2%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
packages/service-library/src/servicelib/utils.py	73.5% <0.0%> (ø)
...re_service_dynamic_sidecar/core/shared_handlers.py	100.0% <ø> (ø)
...r/src/simcore_service_webserver/director_v2_api.py	100.0% <ø> (ø)
...mcore_service_dynamic_sidecar/modules/nodeports.py	21.6% <21.6%> (ø)
.../src/simcore_service_webserver/director_v2_core.py	64.6% <27.7%> (-3.4%)	⬇️
...vice_webserver/projects/projects_nodes_handlers.py	70.2% <30.0%> (-6.3%)	⬇️
.../src/simcore_service_dynamic_sidecar/core/utils.py	80.0% <40.0%> (-2.8%)	⬇️
...re_service_dynamic_sidecar/modules/data_manager.py	46.8% <46.8%> (ø)
..._director_v2/modules/dynamic_sidecar/client_api.py	77.2% <57.4%> (-5.9%)	⬇️
...s/dynamic_sidecar/docker_service_specs/settings.py	75.0% <75.0%> (ø)
... and 35 more