Skip to content

Commit 9be551a

Browse files
GitHKAndrei Neagusandereggpcrespov
authored
Adds dynamic-sidecar osparc service (🏗️actions required) (#2271)
* moved service-sidecar contents * added task to run unittests * added unittests bash commands * added command to run unittests locally * added entry to build dynamic-sidecar * coverage and deploy now await for dynamic-sidecar * renamed service-sidecar to dynamic-sidecar * renaming missing folder * renaming missing fodler part 2 * updating makefile * updating tox * removing tox * added ci codestyle check before tests run * removed unsued targets * fixed error message * added missing test * added codeoowner entry * moved mocked fucntions to the same module * renamed module * updating log warning messages for mocked calls * added some help * clrifying command usage * using verbatim arguments * fixed typo * renived tests * renaming ServiceSidecarSettings to DynamicSidecarSettings * updated name in changelog * fixed ptsv entrypoint * fixed healt endpoint to fail when status is not healthy * fixed route and test * mappign docs on the same route as other services * added api prefix for interested routes * codestyle * container api routes refactored * renamed storage to shared_store * refacted shared_store to use Pydantic models * missed a rename * added makefile entry to generate openapi spec * adding correct mypy file * mypy suggestions * added openapi.json for the service * codestyle also updates openapi.json * added common scripts to enforce codestyles in development and CI * imported statuses from fastapi/starlette definitions * updated docstring * fixed imports, now importing from starlette * moved mark.asyncio * moved act into bash script * added act entry to common.Makefile * added entry for development of dunamic-sidecar * added test to check the spec has was updated * added missing fixture * updated readme * removed unused APIs and renamed existing * added docker-compose config validation * moved settings and shared state to dependencies * minor improvments * updated requirements * added missing service to SERVICES_LIST * Some suggestions (#3) from sanderegg * naming * consistency * shellcheck * proposed refactoring on dynamic sidecar (#4) * removes responses and raise exceptions for errors * Minor typo * Should include standard entrypoint * refactor health and added dependencies * minor * fixes * fixed codestyle * fixed typos * fixed compose API entrypoints and tests * refactored api structure and endpoints * extended utils * adapted new tests * updated API spec * moving routes under the same module * moved tests and updated openapi.json * updated script entryoint name * renamed endpoint * refactor * refactor error raising and staus code for resource not found * regenerated openapi.json spec * codestyle :\ * merged /containers and /containers:inspect * wrong default value * test clenup * updated openapi.json * clarify comment * removed unused API * removed command timeout when posting to createing services via docker-compose * removing unecessary timeouts * forgot to implement it * after the request is accepted the list of container names is returned * removed flag which is no longer used * fixed tests * moved ApplicationHealth and codestyle fixes * added test to check it can run twice the same compos spec with different project-name * fixed codestyle * setting default and documentation to argument * updated oepnapi.json * fixed test to properly work based on workspace * refactoring application models structure * rending applicaiton strucutre similar to other fastapi services * replacing with response.json() * applied codestyle * comitting new updated dependencies * refactoring error codes * clened up exception throwing * restored error emssage Co-authored-by: Andrei Neagu <[email protected]> Co-authored-by: Sylvain <[email protected]> Co-authored-by: Pedro Crespo-Valero <[email protected]>
1 parent d9125d4 commit 9be551a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+2856
-0
lines changed

‎.github/CODEOWNERS

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ Makefile @pcrespov, @sanderegg
2222
/scripts/template-projects/ @odeimaiz, @pcrespov
2323
/services/api-server/ @pcrespov
2424
/services/director*/ @sanderegg, @pcrespov
25+
/services/dynamic-sidecar/ @GitHK
2526
/services/catalog/ @pcrespov, @sanderegg
2627
/services/migration/ @pcrespov
2728
/services/sidecar/ @pcrespov, @mguidon

‎.github/workflows/ci-testing-deploy.yml

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -313,6 +313,55 @@ jobs:
313313
name: codeclimate-${{ github.job }}-coverage
314314
path: codeclimate.${{ github.job }}_coverage.json
315315

316+
unit-test-dynamic-sidecar:
317+
name: "[unit] dynamic-sidecar"
318+
runs-on: ${{ matrix.os }}
319+
strategy:
320+
matrix:
321+
python: [3.6]
322+
os: [ubuntu-20.04]
323+
fail-fast: false
324+
steps:
325+
- uses: actions/checkout@v2
326+
- name: setup docker
327+
run: |
328+
sudo ./ci/github/helpers/setup_docker_compose.bash
329+
./ci/github/helpers/setup_docker_experimental.bash
330+
- name: setup python environment
331+
uses: actions/setup-python@v2
332+
with:
333+
python-version: ${{ matrix.python }}
334+
- name: show system version
335+
run: ./ci/helpers/show_system_versions.bash
336+
- uses: actions/cache@v2
337+
name: getting cached data
338+
with:
339+
path: ~/.cache/pip
340+
key: ${{ runner.os }}-pip-dynamic-sidecar-${{ hashFiles('services/dynamic-sidecar/requirements/ci.txt') }}
341+
restore-keys: |
342+
${{ runner.os }}-pip-dynamic-sidecar-
343+
${{ runner.os }}-pip-
344+
${{ runner.os }}-
345+
- name: install
346+
run: ./ci/github/unit-testing/dynamic-sidecar.bash install
347+
- name: codestyle
348+
run: ./ci/github/unit-testing/dynamic-sidecar.bash codestyle
349+
- name: test
350+
run: ./ci/github/unit-testing/dynamic-sidecar.bash test
351+
- uses: codecov/codecov-action@v1
352+
with:
353+
flags: unittests #optional
354+
- name: prepare codeclimate coverage file
355+
run: |
356+
curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-0.7.0-linux-amd64 > ./cc-test-reporter
357+
chmod +x ./cc-test-reporter && ./cc-test-reporter --version
358+
./cc-test-reporter format-coverage -t coverage.py -o codeclimate.${{ github.job }}_coverage.json coverage.xml
359+
- name: upload codeclimate coverage
360+
uses: actions/upload-artifact@v2
361+
with:
362+
name: codeclimate-${{ github.job }}-coverage
363+
path: codeclimate.${{ github.job }}_coverage.json
364+
316365
unit-test-frontend:
317366
name: "[unit] frontend"
318367
runs-on: ${{ matrix.os }}
@@ -1752,6 +1801,7 @@ jobs:
17521801
unit-test-director,
17531802
unit-test-director-v2,
17541803
unit-test-sidecar,
1804+
unit-test-dynamic-sidecar,
17551805
unit-test-service-integration,
17561806
unit-test-service-library,
17571807
unit-test-models-library,
@@ -1806,6 +1856,7 @@ jobs:
18061856
unit-test-director,
18071857
unit-test-director-v2,
18081858
unit-test-sidecar,
1859+
unit-test-dynamic-sidecar,
18091860
unit-test-frontend,
18101861
unit-test-python-linting,
18111862
unit-test-service-integration,

‎Makefile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ SERVICES_LIST := \
3232
catalog \
3333
director \
3434
director-v2 \
35+
dynamic-sidecar \
3536
migration \
3637
sidecar \
3738
storage \

‎ci/github/unit-testing/dynamic-sidecar.bash

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/bin/bash
2+
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
3+
set -o errexit # abort on nonzero exitstatus
4+
set -o nounset # abort on unbound variable
5+
set -o pipefail # don't hide errors within pipes
6+
IFS=$'\n\t'
7+
8+
install() {
9+
bash ci/helpers/ensure_python_pip.bash;
10+
pushd services/dynamic-sidecar; pip3 install -r requirements/ci.txt -r requirements/_tools.txt; popd;
11+
pip list -v
12+
}
13+
14+
codestyle(){
15+
scripts/codestyle.bash ci simcore_service_dynamic_sidecar services/dynamic-sidecar
16+
}
17+
18+
test() {
19+
pytest --cov=simcore_service_dynamic_sidecar --durations=10 --cov-append \
20+
--color=yes --cov-report=term-missing --cov-report=xml --cov-config=.coveragerc \
21+
-v -m "not travis" services/dynamic-sidecar/tests/unit
22+
}
23+
24+
# Check if the function exists (bash specific)
25+
if declare -f "$1" > /dev/null
26+
then
27+
# call arguments verbatim
28+
"$@"
29+
else
30+
# Show a helpful error
31+
echo "'$1' is not a known function name" >&2
32+
exit 1
33+
fi

‎scripts/act.bash

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/bin/bash
2+
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
3+
set -o errexit
4+
set -o nounset
5+
set -o pipefail
6+
IFS=$'\n\t'
7+
8+
# points to root soruce directory of this project, usually ../../
9+
ROOT_PROJECT_DIR=$1
10+
# name of the job, usually defined in the .github/workflows/ci-testing-deploy.yaml
11+
JOB_TO_RUN=$2
12+
13+
DOCKER_IMAGE_NAME=dind-act-runner
14+
ACT_RUNNER=ubuntu-20.04=catthehacker/ubuntu:act-20.04
15+
ACT_VERSION_TAG=v0.2.20 # from https://github.com/nektos/act/releases
16+
17+
docker build -t $DOCKER_IMAGE_NAME - <<EOF
18+
FROM docker:dind
19+
20+
RUN apk add curl bash
21+
RUN curl -fsSL https://raw.githubusercontent.com/nektos/act/master/install.sh | bash /dev/stdin -d $ACT_VERSION_TAG
22+
23+
WORKDIR /project
24+
25+
CMD /bin/sh -c "act -v -P $ACT_RUNNER -j $JOB_TO_RUN"
26+
EOF
27+
28+
echo "$(pwd)/${ROOT_PROJECT_DIR}"
29+
docker run --rm -it \
30+
-v /var/run/docker.sock:/var/run/docker.sock \
31+
-v "$(pwd)"/"${ROOT_PROJECT_DIR}":/project \
32+
-v "$(pwd)"/ci-logs:/logs \
33+
$DOCKER_IMAGE_NAME

‎scripts/codestyle.bash

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/bin/bash
2+
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
3+
set -o errexit
4+
set -o nounset
5+
set -o pipefail
6+
IFS=$'\n\t'
7+
8+
SRC_DIRECTORY_NAME=${2}
9+
BASE_PATH_DIR=${3-MISSING_DIR}
10+
11+
# used for development (fails on pylint and mypy)
12+
development() {
13+
echo "enforcing codestyle to source_directory=$SRC_DIRECTORY_NAME"
14+
echo "isort"
15+
isort setup.py src/"$SRC_DIRECTORY_NAME" tests
16+
echo "black"
17+
black src/"$SRC_DIRECTORY_NAME" tests/
18+
echo "pylint"
19+
pylint --rcfile=../../.pylintrc src/"$SRC_DIRECTORY_NAME" tests/
20+
echo "mypy"
21+
mypy --ignore-missing-imports --config-file ../../mypy.ini src/"$SRC_DIRECTORY_NAME" tests/
22+
}
23+
24+
# invoked by ci as test (also fails on isort and black)
25+
ci() {
26+
echo "checking codestyle in service=$BASE_PATH_DIR with source_directory=$SRC_DIRECTORY_NAME"
27+
echo "isort"
28+
isort --check setup.py "$BASE_PATH_DIR"/src/"$SRC_DIRECTORY_NAME" "$BASE_PATH_DIR"/tests
29+
echo "black"
30+
black --check "$BASE_PATH_DIR"/src/"$SRC_DIRECTORY_NAME" "$BASE_PATH_DIR"/tests
31+
echo "pylint"
32+
pylint --rcfile=.pylintrc "$BASE_PATH_DIR"/src/"$SRC_DIRECTORY_NAME" "$BASE_PATH_DIR"/tests
33+
echo "mypy"
34+
mypy --config-file mypy.ini --ignore-missing-imports "$BASE_PATH_DIR"/src/"$SRC_DIRECTORY_NAME" "$BASE_PATH_DIR"/tests
35+
}
36+
37+
# Allows to call a function based on arguments passed to the script
38+
"$@"

‎scripts/common.Makefile

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,16 @@ code-analysis: $(REPO_BASE_DIR)/.codeclimate.yml ## runs code-climate analysis
131131
@-rm $(CURDIR)/.codeclimate.yml
132132

133133

134+
.PHONY: codestyle
135+
codestyle: ## enforces codestyle (isort & black) finally runs pylint & mypy
136+
@$(SCRIPTS_DIR)/codestyle.bash development $(shell basename "${SRC_DIR}")
137+
138+
.PHONY: github-workflow-job
139+
github-workflow-job: ## runs a github workflow job using act locally, run using "make github-workflow-job job=JOB_NAME"
140+
# running job "${job}"
141+
$(SCRIPTS_DIR)/act.bash ../.. ${job}
142+
143+
134144
.PHONY: version-patch version-minor version-major
135145
version-patch: ## commits version with bug fixes not affecting the cookiecuter config
136146
$(_bumpversion)

‎services/docker-compose-build.yml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,22 @@ services:
124124
org.label-schema.vcs-url: "${VCS_URL}"
125125
org.label-schema.vcs-ref: "${VCS_REF}"
126126

127+
dynamic-sidecar:
128+
image: local/dynamic-sidecar:${BUILD_TARGET:?build_target_required}
129+
build:
130+
context: ../
131+
dockerfile: services/dynamic-sidecar/Dockerfile
132+
cache_from:
133+
- local/dynamic-sidecar:${BUILD_TARGET:?build_target_required}
134+
- ${DOCKER_REGISTRY:-itisfoundation}/dynamic-sidecar:cache
135+
- ${DOCKER_REGISTRY:-itisfoundation}/dynamic-sidecar:${DOCKER_IMAGE_TAG:-latest}
136+
target: ${BUILD_TARGET:?build_target_required}
137+
labels:
138+
org.label-schema.schema-version: "1.0"
139+
org.label-schema.build-date: "${BUILD_DATE}"
140+
org.label-schema.vcs-url: "${VCS_URL}"
141+
org.label-schema.vcs-ref: "${VCS_REF}"
142+
127143
storage:
128144
image: local/storage:${BUILD_TARGET:?build_target_required}
129145
build:

‎services/dynamic-sidecar/.env-devel

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Environment used to configure storage services
2+
#
3+
# - To expose in cmd: export $(grep -v '^#' .env-devel | xargs -0)
4+
#
5+
6+
# environs in Dockerfile ----------------
7+
SC_BOOT_MODE=local-development
8+
9+
10+
# service specific required vars
11+
DYNAMIC_SIDECAR_compose_namespace=dev-namespace
12+
DYNAMIC_SIDECAR_docker_compose_down_timeout=15

‎services/dynamic-sidecar/CHANGELOG.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Changelog
2+
All notable changes to this service will be documented in this file.
3+
4+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
5+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6+
7+
## [Unreleased]
8+
9+
## [0.0.1] - 2021-04-14
10+
### Added
11+
- First working version of the service
12+
- FastAPI based service
13+
- all spawned services and networks are removed when receiving `SIGTERM`

0 commit comments

Comments
 (0)