Fixes on catalog/services section of the web-API (#2151)

Requests in the catalog/services section of the web-API are validated in the webserver and forwarded to the catalog. The responses of the catalog are then again processed in the webserver (adds convenience values for the units) and retransmitted back to the front-end.

Related to #2141: Processing of metadata in pipelines: units

* catalog OAS published under api/specs/catalog

* Adds constraint on uvloop (dropping support for 3.6!)

* full upgrade of catalog reqs

* Minor cleanup .env file

* Increases test coverage

* Cleanup tests: reorganize and delete old
Adds tests for models

* Added examples for testing

* Adds a layer in the web-server that extends response of catalog

* Serialization done with orjson
Adds tests on replacement

* fixes fixtures

* Fixes example

* Fixes catalog OAS failure

* Prunes defaultValue in output ports

* Change title of web API

* Catalog entrypoints response model excludes None fields

* @sanderegg review: restored return annotations and added some more for redemption :-)

* @sanderegg review: policy drops only unset values instead of nulls for responses
Units are formatted ONLY when provided

* Fixes public-api test

Author: pcrespov

.env-devel

@@ -1,21 +1,25 @@
 #
-# - Keep it alfphabetical order and grouped by prefix
+# - Keep it alfphabetical order and grouped by prefix [see vscode cmd: Sort Lines Ascending]
 # - To expose:  export $(grep -v '^#' .env | xargs -0)
 #
+
 API_SERVER_DEV_FEATURES_ENABLED=0
 
 BF_API_KEY=none
 BF_API_SECRET=none
 
+DIRECTOR_REGISTRY_CACHING_TTL=900
+DIRECTOR_REGISTRY_CACHING=True
+
 POSTGRES_DB=simcoredb
 POSTGRES_ENDPOINT=postgres:5432
 POSTGRES_HOST=postgres
 POSTGRES_PASSWORD=adminadmin
 POSTGRES_PORT=5432
 POSTGRES_USER=scu
 
-RABBIT_HOST=rabbit
 RABBIT_CHANNELS={"log": "comp.backend.channels.log", "instrumentation": "comp.backend.channels.instrumentation"}
+RABBIT_HOST=rabbit
 RABBIT_PASSWORD=adminadmin
 RABBIT_PORT=5672
 RABBIT_USER=admin
@@ -28,12 +32,10 @@ REGISTRY_PW=adminadmin
 REGISTRY_SSL=True
 REGISTRY_URL=registry.osparc-master.speag.com
 REGISTRY_USER=admin
-DIRECTOR_REGISTRY_CACHING=True
-DIRECTOR_REGISTRY_CACHING_TTL=900
 
+# NOTE: 172.17.0.1 is the docker0 interface, which redirect from inside a container onto the host network interface.
 S3_ACCESS_KEY=12345678
 S3_BUCKET_NAME=simcore
-# 172.17.0.1 is the docker0 interface, which redirect from inside a container onto the host network interface.
 S3_ENDPOINT=172.17.0.1:9001
 S3_SECRET_KEY=12345678
 S3_SECURE=0
@@ -51,25 +53,25 @@ TRACING_ZIPKIN_ENDPOINT=http://jaeger:9411
 
 TRAEFIK_SIMCORE_ZONE=internal_simcore_stack
 
+# NOTE: WEBSERVER_SESSION_SECRET_KEY = $(python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key())")
 WEBSERVER_DEV_FEATURES_ENABLED=0
 WEBSERVER_HOST=webserver
 WEBSERVER_LOGIN_REGISTRATION_CONFIRMATION_REQUIRED=0
 WEBSERVER_LOGIN_REGISTRATION_INVITATION_REQUIRED=0
-# python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key())"
-WEBSERVER_SESSION_SECRET_KEY=REPLACE ME with a key of at least length 32.
-WEBSERVER_STUDIES_ACCESS_ENABLED=0
+WEBSERVER_FEEDBACK_FORM_URL=https://docs.google.com/forms/d/e/1FAIpQLSe232bTigsM2zV97Kjp2OhCenl6o9gNGcDFt2kO_dfkIjtQAQ/viewform?usp=sf_link
+WEBSERVER_FOGBUGZ_LOGIN_URL=https://z43.manuscript.com/login
+WEBSERVER_FOGBUGZ_NEWCASE_URL=https://z43.manuscript.com/f/cases/new?command=new&pg=pgEditBug&ixProject=45&ixArea=449
+WEBSERVER_GARBAGE_COLLECTION_INTERVAL_SECONDS=30
+WEBSERVER_MANUAL_EXTRA_URL=https://itisfoundation.github.io/osparc-manual-z43/
+WEBSERVER_MANUAL_MAIN_URL=http://docs.osparc.io/
+WEBSERVER_PROMETHEUS_API_VERSION=v1
 WEBSERVER_PROMETHEUS_HOST=http://prometheus
 WEBSERVER_PROMETHEUS_PORT=9090
-WEBSERVER_PROMETHEUS_API_VERSION=v1
 WEBSERVER_RESOURCES_DELETION_TIMEOUT_SECONDS=900
-WEBSERVER_GARBAGE_COLLECTION_INTERVAL_SECONDS=30
-WEBSERVER_MANUAL_MAIN_URL=http://docs.osparc.io/
-WEBSERVER_MANUAL_EXTRA_URL=https://itisfoundation.github.io/osparc-manual-z43/
-WEBSERVER_FOGBUGZ_LOGIN_URL=https://z43.manuscript.com/login
-WEBSERVER_FOGBUGZ_NEWCASE_URL=https://z43.manuscript.com/f/cases/new?command=new&pg=pgEditBug&ixProject=45&ixArea=449
 WEBSERVER_S4L_FOGBUGZ_NEWCASE_URL=https://z43.manuscript.com/f/cases/new?command=new&pg=pgEditBug&ixProject=45&ixArea=458
+WEBSERVER_SESSION_SECRET_KEY=REPLACE ME with a key of at least length 32.
+WEBSERVER_STUDIES_ACCESS_ENABLED=0
 WEBSERVER_TIS_FOGBUGZ_NEWCASE_URL=https://z43.manuscript.com/f/cases/new?command=new&pg=pgEditBug&ixProject=45&ixArea=459
-WEBSERVER_FEEDBACK_FORM_URL=https://docs.google.com/forms/d/e/1FAIpQLSe232bTigsM2zV97Kjp2OhCenl6o9gNGcDFt2kO_dfkIjtQAQ/viewform?usp=sf_link
 WEBSERVER_VIEWER_RAWGRAPH_VERSION=2.11.1
 WEBSERVER_VIEWER_SIM4LIFE_VERSION=1.0.29
 

api/specs/webserver/openapi-catalog.yaml

@@ -66,18 +66,18 @@ paths:
         "422":
           description: Validation Error
 
-  /catalog/services:
+  catalog_services:
     get:
       tags:
         - catalog
       summary: List Services
-      operationId: list_catalog_services
+      operationId: list_services_handler
       responses:
         "200":
           description: Returns list of services from the catalog
         default:
           $ref: "./openapi.yaml#/components/responses/DefaultErrorResponse"
-  /catalog/services/{service_key}/{service_version}:
+  catalog_services_service_key_service_version:
     parameters:
       - in: path
         name: service_key
@@ -97,7 +97,7 @@ paths:
       tags:
         - catalog
       summary: Get Service
-      operationId: get_catalog_service
+      operationId: get_service_handler
       responses:
         "200":
           description: Returns service
@@ -107,7 +107,7 @@ paths:
       tags:
         - catalog
       summary: Update Service
-      operationId: update_catalog_service
+      operationId: update_service_handler
       requestBody:
         content:
           application/json:

api/specs/webserver/openapi.yaml

@@ -1,8 +1,8 @@
 openapi: 3.0.0
 info:
-  title: "osparc-simcore RESTful API"
+  title: "osparc-simcore web API"
   version: 0.6.0
-  description: "RESTful API designed for web clients"
+  description: "API designed for the front-end app"
   contact:
     name: IT'IS Foundation
     email: [email protected]
@@ -218,11 +218,11 @@ paths:
     $ref: "./openapi-catalog.yaml#/paths/~1catalog~1dags"
   /catalog/dags/{dag_id}:
     $ref: "./openapi-catalog.yaml#/paths/~1catalog~1dags~1{dag_id}"
+
   /catalog/services:
-    $ref: "./openapi-catalog.yaml#/paths/~1catalog~1services"
+    $ref: "./openapi-catalog.yaml#/paths/catalog_services"
   /catalog/services/{service_key}/{service_version}:
-    $ref: "./openapi-catalog.yaml#/paths/~1catalog~1services~1{service_key}~1{service_version}"
-
+    $ref: "./openapi-catalog.yaml#/paths/catalog_services_service_key_service_version"
   /catalog/services/{service_key}/{service_version}/inputs:
     $ref: "./openapi-catalog.yaml#/paths/catalog_services_service_key_service_version_inputs"
   /catalog/services/{service_key}/{service_version}/inputs/{input_key}:

api/tests/conftest.py

@@ -10,98 +10,107 @@
 from pathlib import Path
 
 import pytest
-
 from utils import is_json_schema, load_specs
 
 log = logging.getLogger(__name__)
 
 # Conventions
-COMMON = 'common'
-OPENAPI_MAIN_FILENAME = 'openapi.yaml'
+COMMON = "common"
+OPENAPI_MAIN_FILENAME = "openapi.yaml"
 
 current_dir = Path(sys.argv[0] if __name__ == "__main__" else __file__).resolve().parent
 
 
-@pytest.fixture(scope='session')
-def this_repo_root_dir():
+@pytest.fixture(scope="session")
+def this_repo_root_dir() -> Path:
     root_dir = current_dir.parent.parent
     assert root_dir
     assert any(root_dir.glob(".git"))
     return root_dir
 
 
-@pytest.fixture(scope='session')
-def api_specs_dir():
+@pytest.fixture(scope="session")
+def api_specs_dir() -> Path:
     return current_dir.parent / "specs"
 
 
 @pytest.fixture(scope="session")
-def webserver_api_dir(api_specs_dir):
+def webserver_api_dir(api_specs_dir) -> Path:
     return api_specs_dir / "webserver"
 
-@pytest.fixture(scope='session')
+
+@pytest.fixture(scope="session")
 def api_specs_info(api_specs_dir):
     """
-        Returns a namedtuple with info on every
+    Returns a namedtuple with info on every
     """
-    service_dirs = [d for d in api_specs_dir.iterdir() if d.is_dir() and not d.name.endswith(COMMON)]
+    service_dirs = [
+        d for d in api_specs_dir.iterdir() if d.is_dir() and not d.name.endswith(COMMON)
+    ]
 
-    info_cls = namedtuple("ApiSpecsInfo", "service version openapi_path url_path".split())
+    info_cls = namedtuple(
+        "ApiSpecsInfo", "service version openapi_path url_path".split()
+    )
     info = []
     for srv_dir in service_dirs:
-        version_dirs = [d for d in srv_dir.iterdir() if d.is_dir() and not d.name.endswith(COMMON)]
+        version_dirs = [
+            d for d in srv_dir.iterdir() if d.is_dir() and not d.name.endswith(COMMON)
+        ]
         for ver_dir in version_dirs:
             openapi_path = ver_dir / OPENAPI_MAIN_FILENAME
             if openapi_path.exists():
-                info.append( info_cls(
-                    service=srv_dir.name,
-                    version=ver_dir.name,
-                    openapi_path=openapi_path,
-                    url_path=relpath(openapi_path, srv_dir) # ${version}/openapi.yaml
-                ))
+                info.append(
+                    info_cls(
+                        service=srv_dir.name,
+                        version=ver_dir.name,
+                        openapi_path=openapi_path,
+                        url_path=relpath(
+                            openapi_path, srv_dir
+                        ),  # ${version}/openapi.yaml
+                    )
+                )
     # https://yarl.readthedocs.io/en/stable/api.html#yarl.URL
     # [scheme:]//[user[:password]@]host[:port][/path][?query][#fragment]
     return info
 
 
 @pytest.fixture(scope="session")
 def all_api_specs_tails(api_specs_dir):
-    """ Returns openapi/jsonschema spec files path relative to specs_dir
-
-    """
+    """Returns openapi/jsonschema spec files path relative to specs_dir"""
     return _all_api_specs_tails_impl(api_specs_dir)
 
+
 def _all_api_specs_tails_impl(api_specs_dir):
     tails = []
-    for fpath in chain(*[api_specs_dir.rglob(wildcard) for wildcard in ("*.json", "*.y*ml")]):
+    for fpath in chain(
+        *[api_specs_dir.rglob(wildcard) for wildcard in ("*.json", "*.y*ml")]
+    ):
         tail = relpath(fpath, api_specs_dir)
-        tails.append(Path(tail) )
+        tails.append(Path(tail))
     return tails
 
 
 def list_openapi_tails():
-    """ Returns relative path to all non-jsonschema (i.e. potential openapi)
+    """Returns relative path to all non-jsonschema (i.e. potential openapi)
 
-        SEE api_specs_tail to get one at a time
+    SEE api_specs_tail to get one at a time
     """
     tails = []
     specs_dir = current_dir.parent / "specs"
     for tail in _all_api_specs_tails_impl(specs_dir):
-        specs = load_specs( specs_dir / tail)
+        specs = load_specs(specs_dir / tail)
         if not is_json_schema(specs):
-            tails.append( str(tail) )
+            tails.append(str(tail))
     return tails
 
 
-@pytest.fixture(scope="session",
-                params=list_openapi_tails()
-                )
+@pytest.fixture(scope="session", params=list_openapi_tails())
 def api_specs_tail(request, api_specs_dir):
-    """ Returns api specs file path relative to api_specs_dir
+    """Returns api specs file path relative to api_specs_dir
 
-        NOTE: this is a parametrized fixture that
-          represents one api-specs tail at a time!
-        NOTE: as_str==True, so it gets printed
+    NOTE: this is a parametrized fixture that
+      represents one api-specs tail at a time!
+    NOTE: as_str==True, so it gets printed
     """
     specs_tail = request.param
     assert exists(api_specs_dir / specs_tail)

packages/models-library/src/models_library/projects_nodes.py

@@ -125,7 +125,6 @@ class Node(BaseModel):
     run_hash: Optional[str] = Field(
         None,
         description="the hex digest of the resolved inputs +outputs hash at the time when the last outputs were generated",
-        examples=["a4337bc45a8fc544c03f52dc550cd6e1e87021bc896588bd79e901e2"],
         alias="runHash",
     )
 
@@ -158,7 +157,6 @@ class Node(BaseModel):
     parent: Optional[NodeID] = Field(
         None,
         description="Parent's (group-nodes') node ID s. Used to group",
-        examples=["nodeUUid1", "nodeUuid2"],
     )
 
     # NOTE: use projects_ui.py

packages/models-library/src/models_library/services.py

@@ -363,10 +363,13 @@ class ServiceAccessRights(BaseModel):
 
 
 class ServiceMetaData(ServiceCommonData):
-    # for a partial update all members must be Optional
+    # Overrides all fields of ServiceCommonData:
+    #    - for a partial update all members must be Optional
     name: Optional[str]
     thumbnail: Optional[HttpUrl]
     description: Optional[str]
+
+    # user-defined metatada
     classifiers: Optional[List[str]]
     quality: Dict[str, Any] = {}
 

packages/service-library/tests/conftest.py

@@ -4,10 +4,9 @@
 
 import sys
 from pathlib import Path
+from typing import Dict
 
 import pytest
-import yaml
-
 import servicelib
 from servicelib.openapi import create_openapi_specs
 
@@ -18,14 +17,14 @@ def here():
 
 
 @pytest.fixture(scope="session")
-def package_dir():
+def package_dir() -> Path:
     pdir = Path(servicelib.__file__).resolve().parent
     assert pdir.exists()
     return pdir
 
 
 @pytest.fixture(scope="session")
-def osparc_simcore_root_dir(here):
+def osparc_simcore_root_dir(here) -> Path:
     root_dir = here.parent.parent.parent.resolve()
     assert root_dir.exists(), "Is this service within osparc-simcore repo?"
     assert any(root_dir.glob("packages/service-library")), (
@@ -35,13 +34,13 @@ def osparc_simcore_root_dir(here):
 
 
 @pytest.fixture
-def petstore_spec_file(here):
+def petstore_spec_file(here) -> Path:
     filepath = here / "data/oas3/petstore.yaml"
     assert filepath.exists()
     return filepath
 
 
 @pytest.fixture
-async def petstore_specs(loop, petstore_spec_file):
+async def petstore_specs(loop, petstore_spec_file) -> Dict:
     specs = await create_openapi_specs(petstore_spec_file)
     return specs

requirements/constraints.txt

@@ -36,3 +36,7 @@ idna<3,>=2.5
 
 # vulnerability https://github.com/advisories/GHSA-rhm9-p9w5-fwm7  Feb.2021
 cryptography>=3.3.2
+
+
+# constraint since https://github.com/MagicStack/uvloop/releases/tag/v0.15.0: drops support for 3.5/3.6  Feb.2021
+uvloop<0.15.0 ; python_version < '3.7'

scripts/common-service.Makefile

@@ -70,7 +70,7 @@ shell: ## runs shell inside $(APP_NAME) container
 
 .PHONY: tail
 tail: ## tails log of $(APP_NAME) container
-	docker logs --follow $(shell docker ps -f "name=$(APP_NAME)*" --format {{.ID}}) 2>&1
+	docker logs --follow $(shell docker ps --filter "name=$(APP_NAME)*" --format {{.ID}}) 2>&1
 
 
 .PHONY: stats

services/api-server/tests/unit/conftest.py

@@ -75,15 +75,15 @@ def project_env_devel_environment(project_env_devel_dict, monkeypatch):
 
 
 @pytest.fixture(scope="session")
-def project_slug_dir():
+def project_slug_dir() -> Path:
     folder = current_dir.parent.parent
     assert folder.exists()
     assert any(folder.glob("src/simcore_service_api_server"))
     return folder
 
 
 @pytest.fixture(scope="session")
-def package_dir():
+def package_dir() -> Path:
     """Notice that this might be under src (if installed as edit mode)
     or in the installation folder
     """