Skip to content

Create server rest API #186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
10 tasks done
pcrespov opened this issue Aug 15, 2018 · 2 comments
Closed
10 tasks done

Create server rest API #186

pcrespov opened this issue Aug 15, 2018 · 2 comments
Assignees
Milestone

Comments

@pcrespov
Copy link
Member

pcrespov commented Aug 15, 2018

Focus: Unify services APIs paths and models (both architecture and development tools)

  • authentication api (specs only) (required by user auth business logic #200)
  • Health-check api + handler (logic), returns whether the service runs, DB connected, etc..
  • director access using its client-sdk (needs separate files requires Create director API #185)
  • apidoc should only expose entrypoints (no /qxapp...)
  • create ORM classes and init db
  • Minimal routes
    • api/v1/$auth
    • api/v1/openapi.json → {open api}: /apidoc/swagger.yaml?spec=/v1
    • api/v1/projects endpoint
  • upgrade Dockerfile to include .venv and pip install 3rd/2nd parties
  • generate python/js client-sdk for this service: webserver-sdk

child of #19 and related with issue #148
blocks #188

@pcrespov pcrespov added this to the Libra milestone Aug 15, 2018
@pcrespov pcrespov self-assigned this Aug 15, 2018
@pcrespov pcrespov mentioned this issue Aug 24, 2018
5 tasks
pcrespov pushed a commit to pcrespov/osparc-simcore that referenced this issue Aug 28, 2018
pcrespov pushed a commit to pcrespov/osparc-simcore that referenced this issue Aug 30, 2018
commit 0df033b
Author: Pedro Crespo <[email protected]>
Date:   Thu Aug 30 14:21:44 2018 +0200

    Review feedback: removed unnecessary config sections

commit bab7831
Author: Pedro Crespo <[email protected]>
Date:   Thu Aug 30 13:59:07 2018 +0200

    Fixes missing dlls

commit c0bc7d6
Author: Pedro Crespo <[email protected]>
Date:   Thu Aug 30 11:53:41 2018 +0200

    Fixes access to docker socket:
     - cleanup Dockerfile
     - minor doc and logs

commit 5b6f4c7
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 17:47:13 2018 +0200

    Minor cleanup and doc

commit 43f7305
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:52:30 2018 +0200

    Minor change in web server

commit 7f07022
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:52:16 2018 +0200

    sidecar: starts as root (via entrypoint) but runs as scu (non-root)

commit 90359c4
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:51:32 2018 +0200

    Fixes sidecar: temporary fix to connection to other services

commit 7619671
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:44:42 2018 +0200

    Fixes host in server running inside a docker

commit bdbf078
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:43:56 2018 +0200

    Separating maintenance services

commit 83e492d
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 13:46:55 2018 +0200

    Fixes on web server:
     - removed usused cs_s4l from config
     - added env for rabbit

commit 8fee402
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 11:48:46 2018 +0200

    Fixes on webserver: - Tmp  solution for production target based on production-build stage -  Normalized naming of package and console script

commit a5d8220
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 22:06:59 2018 +0200

    Implements entry in ITISFoundation#186: - upgrades Dockerfile with .venv and pip installing 3rd parties

commit 3c55bf3
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 18:46:57 2018 +0200

    Setting up boot for web service and forgot configs

commit 8003744
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 18:40:12 2018 +0200

    Adding config data and retreiving it as resource

commit 94de3d7
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 10:58:02 2018 +0200

    Minor changes in docker after discussion with MaG

commit e663a1e
Author: Pedro Crespo <[email protected]>
Date:   Mon Aug 27 22:08:38 2018 +0200

    Updated ownership

commit bf486b3
Author: Pedro Crespo <[email protected]>
Date:   Mon Aug 27 21:53:12 2018 +0200

    Fixes linting errors and added notes

commit 752c7bf
Author: Pedro Crespo <[email protected]>
Date:   Mon Aug 27 19:49:39 2018 +0200

    WIP ITISFoundation#198: non-root user
     - all volumes bound to /home/scu
     - All modules pip installed (in dev w/ edit mode).
     - All files produce in dev mode on bound volumes are not deletable
     - Production stage is further optimized by taking only venv from base
     - Fixes sidecar access to input/output/log volumes

commit ec755bc
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 21:53:13 2018 +0200

    WIP ITISFoundation#198:
     - added a boot.sh script to start celery worker
     - restructured package submodules. Otherwise got error (see comments in ITISFoundation#198)
     -

commit 9bbfbde
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 21:52:12 2018 +0200

    Minor update

commit 53c7707
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 20:29:26 2018 +0200

    Fixes import failure of trafaret and updages vs sqalquemy

commit a27a279
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 20:24:49 2018 +0200

    WIP ITISFoundation#198: setup for sidecar
     - created setup and requirements for dev&prod
     - installs package in docker (instead of reference via sys.path). this way all third-parties are installed.
     - minor cleanup of code

commit bbc7cb6
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 20:22:31 2018 +0200

    Minor: sync req versions
@pcrespov pcrespov mentioned this issue Aug 30, 2018
10 tasks
pcrespov pushed a commit to pcrespov/osparc-simcore that referenced this issue Aug 30, 2018
pcrespov pushed a commit to pcrespov/osparc-simcore that referenced this issue Aug 30, 2018
pcrespov pushed a commit to pcrespov/osparc-simcore that referenced this issue Aug 30, 2018
pcrespov added a commit that referenced this issue Aug 31, 2018
## What do these changes do?

Upgrades ``sidecar`` setup and module structure and fixes some issues.

- [x] Missing setup.py
- [x] Missing requirement-dev.txt
- [x] Non-root user in docker.
- [x] venv in docker
- [x] rearrange package structure: ``main.py``, ``tasks.py``,  ``celery.py`` and ``utils.py``
- [x] entrypoint script to start as ``root`` and run as ``scu``

Extra fixes on ``webserver``
- [x] retreived config files as resources
- [x] modified Dockerfile: should replicate the way we test in source
    - keeps same relative folder structure
    - all 3erd, 2nd and main packages install in a venv
    - runs as a non-root user
    - boot script

## Related issue number
- closes #198 

----

* Fixes typos in client after bad merge

* Minor: sync req versions

* WIP #198: setup for sidecar
 - created setup and requirements for dev&prod
 - installs package in docker (instead of reference via sys.path). this way all third-parties are installed.
 - minor cleanup of code

* Fixes import failure of trafaret and updages vs sqalquemy

* Minor update

* WIP #198:
 - added a boot.sh script to start celery worker
 - restructured package submodules. Otherwise got error (see comments in #198)
 -

* WIP #198: non-root user
 - all volumes bound to /home/scu
 - All modules pip installed (in dev w/ edit mode).
 - All files produce in dev mode on bound volumes are not deletable
 - Production stage is further optimized by taking only venv from base
 - Fixes sidecar access to input/output/log volumes

* Fixes linting errors and added notes

* Updated ownership

* Minor changes in docker after discussion with MaG

* Adding config data and retreiving it as resource

* Setting up boot for web service and forgot configs

* Implements entry in #186: - upgrades Dockerfile with .venv and pip installing 3rd parties

* Fixes on webserver: - Tmp  solution for production target based on production-build stage -  Normalized naming of package and console script

* Fixes on web server:
 - removed usused cs_s4l from config
 - added env for rabbit

* Separating maintenance services

* Fixes host in server running inside a docker

* Fixes sidecar: temporary fix to connection to other services

* sidecar: starts as root (via entrypoint) but runs as scu (non-root)

* Minor change in web server

* Minor cleanup and doc

* Fixes access to docker socket:
 - cleanup Dockerfile
 - minor doc and logs

* Fixes missing dlls

* Review feedback: removed unnecessary config sections

* Fixed volume map between sidecar and workcar containers
@sanderegg
Copy link
Member

  • The server shall not stop starting when there is an issue in accessing an external service (such as database,...). It should report it in a healthcheck

pcrespov added a commit to pcrespov/osparc-simcore that referenced this issue Sep 5, 2018
@pcrespov
Copy link
Member Author

pcrespov commented Sep 6, 2018

bug

  • Makes test_login[pyloop] fail
  • Cannot use security features with aiohttp_apiset module.
  • swagger pass the request argument as key-argument adn aiohttp_security middleware expects it as last positional argument
  File "/home/crespo/devp/osparc-simcore/.venv/lib/python3.6/site-packages/aiohttp_session/__init__.py", line 149, in factory
    response = await handler(request)
  File "/home/crespo/devp/osparc-simcore/.venv/lib/python3.6/site-packages/aiohttp_apiset/middlewares.py", line 96, in process
    response = await handler(request)
  File "/home/crespo/devp/osparc-simcore/services/web/server/src/simcore_service_webserver/rest/middlewares.py", line 12, in handle_errors
    response = await handler(request)
  File "/home/crespo/devp/osparc-simcore/.venv/lib/python3.6/site-packages/aiohttp_apiset/swagger/route.py", line 90, in handler
    response = await self._handler(**parameters)
  File "/home/crespo/devp/osparc-simcore/.venv/lib/python3.6/site-packages/aiohttp_security/api.py", line 136, in wrapped
    request = args[-1]

options

         def wrapped(*args, **kwargs):
            request = kwargs.get('request',  args[-1] if args else None)
            if not isinstance(request, web.BaseRequest):
                msg = ("Incorrect decorator usage. "
                       "Expecting `def handler(request)` "
                       "or `def handler(self, request)`.")
                raise RuntimeError(msg)

pcrespov added a commit that referenced this issue Sep 25, 2018
- [x] adds rest-api following same procedure as in director (see #185)
- [x] wait director integration before merging (requires #185)
- [x] webserver-sdk 
- [x] mockup logic (business logic will be added later)
- [x] adds infrastructure for testing
- [x] sessions, security, configuration, resources (draft)


commit c0bc7d6
Author: Pedro Crespo <[email protected]>
Date:   Thu Aug 30 11:53:41 2018 +0200

    Fixes access to docker socket:
     - cleanup Dockerfile
     - minor doc and logs

commit 7f07022
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:52:16 2018 +0200

    sidecar: starts as root (via entrypoint) but runs as scu (non-root)

commit 90359c4
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:51:32 2018 +0200

    Fixes sidecar: temporary fix to connection to other services

commit 7619671
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 15:44:42 2018 +0200

    Fixes host in server running inside a docker

commit 83e492d
Author: Pedro Crespo <[email protected]>
Date:   Wed Aug 29 13:46:55 2018 +0200

    Fixes on web server:
     - removed usused cs_s4l from config
     - added env for rabbit

commit a5d8220
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 22:06:59 2018 +0200

    Implements entry in #186: - upgrades Dockerfile with .venv and pip installing 3rd parties

commit 3c55bf3
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 18:46:57 2018 +0200

    Setting up boot for web service and forgot configs

commit 8003744
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 18:40:12 2018 +0200

    Adding config data and retrieving it as resource

commit 94de3d7
Author: Pedro Crespo <[email protected]>
Date:   Tue Aug 28 10:58:02 2018 +0200

    Minor changes in docker after discussion with MaG


commit 752c7bf
Author: Pedro Crespo <[email protected]>
Date:   Mon Aug 27 19:49:39 2018 +0200

    WIP #198: non-root user
     - all volumes bound to /home/scu
     - All modules pip installed (in dev w/ edit mode).
     - All files produce in dev mode on bound volumes are not deletable
     - Production stage is further optimized by taking only venv from base
     - Fixes sidecar access to input/output/log volumes

commit ec755bc
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 21:53:13 2018 +0200

    WIP #198:
     - added a boot.sh script to start celery worker
     - restructured package submodules. Otherwise got error (see comments in #198)
     -

commit a27a279
Author: Pedro Crespo <[email protected]>
Date:   Fri Aug 24 20:24:49 2018 +0200

    WIP #198: setup for sidecar
     - created setup and requirements for dev&prod
     - installs package in docker (instead of reference via sys.path). this way all third-parties are installed.
     - minor cleanup of code

* is#186 web: Normalized package name and added new placeholders for submodules
* is#186 web: Normalized package name and added new placeholders for submodules
* is#186 web: fix import in docker context
 - temporarily a thick production stage
 - boot and entrypoint sh to .docker/
- separated middlewares, routing and setup
- merged api submodule into rest - removed aiohttp swagger since already in apiset - split rest/handlers
- web api: moved routings into separate functions
- Allowed disabling services from configuration
- Fixes cyclic import rest.routing -> rest.settings
- Fixes fix bug1 in issue #186 using a decorator
- Testing authentication
- added a light/fast test config
- simple test unauthorized access
- imp#186: Route to retrieve openapi specs json /v1/oas
- Fixes test_rest to access to api specs.
- Fixes coverage failure reported in travis
- Simplified resources functionality and updated tests
- Squashed PR 187 before merged to master.
- Merging bulk changes
- Director with openapi that includes new data model.
- Only missing is adapting business loging to new data model (see issue #213)
- Added test to check openapi specs
- Change first oas.server to overcome client-sdk limitation
- Added autogenerated code for webserver-sdk. Added some ignores
@pcrespov pcrespov modified the milestones: Libra, Picon Sep 28, 2018
@pcrespov pcrespov mentioned this issue Oct 3, 2018
3 tasks
pcrespov added a commit that referenced this issue Oct 29, 2018
- Changed apis/ skeleton. Separated tests from apis. 
- Improved testing apihub and api/specs
- Moving webserver api (will still be duplicated in webserver folder for a while)

connected to #241  and  #186

* servicelib: systematic app keys naming
* Added test to validate and create specs from apihub
* Renamed apis -> api and added subfolder specs
Adapted apihub docker and tests
Adapted travis and makefile
Adapted api/tests and extended
* Parametrized tests per file instead of having all files within a single test
Makes file tests independent and make errors more visible
* Assigned owners to api/
* webserver API: copy (and fixes) from services/web/server/src/*/oas3
* Fixes issues with individual openapi schemas
 - created parametrized fixture to loop all openapi spec files
 - refactor converter to process all files and output in a tmp folder. This solves the cross-references problem
@ghost ghost removed the in progress label Nov 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants