WIP: Fix/integration (#6)

* Adds pip workaround as in  ITISFoundation/osparc-simcore#540
Cleans up Dockerfile target `cache`

* Flipped position of 'development' target in Dockerfile to speed up image build of production targets during CI. Now development is a bit slower

* Adds fixes for .pipcache workaround so that it does not fail in devmode
 Changes to be committed:
	modified:   {{cookiecutter.project_slug}}/Dockerfile

* renamed integration -> _osparc-simcore folder

* Fixes typo in README rendering
Minor

* Uninstalls in production stage all dev tools

* output folder has osparc-simcore-like folder structure

* Minor: alias target in makefile and added pre-gen hook as sample

* Fixes test failure

* to log or not to logger ...?

* Added health-check command and script for service

* Minors in Makefile

* cleaned up template Makefile and added usage in README.md

* Fixes setup naming

* Adds setup_swagger to display API documentation

* Added exceptions pckage

* Added microbadger badges for images

* Added banners for released docker images

* updated healtcheck

* Upgrading Dockerfile to deal with non-root users (ITISFoundation/osparc-simcore#90)

* Updated entrypoint and boot as well as docker-compose and .dockerignor

* Updated doc to underline new features

* Adapting travis to run docker tests

* Added make target to build prod docker

* Disabled docker-build

* Minor fixes

* Fixing app console path

* Simplified PR message

* Fixes on docker configs after creating webserver and storage services

* minor fixes and doc
added folder for integration tests WIP

* Disabled client-sdk

* Moved client-sdk and integration tests to keep folder

* Renamed as 'stub'

* Cleaned up cookiecutter attributes list

* Removed all references to docker in the Makefile

* fix exit code

* Fixes development mode in windows where docker mounts all volumes as root

* literal user name instead of variable

* Changend environment variables prefix
Simplified healthcheck

* Minor linter

* Fix issue when docker group name already exists

* REview: disabled all todos and other minor

* Added test to check for tags in all generated files

* Review: reduced timeout

* Cleanedup readme

* Fixes test and minor changes

* fixes tests

* Review: commented out installed packages in Dockerfile

* fixed TODOs

Author: pcrespov

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,26 +1,32 @@
 ---
-name: Feature request
-about: Suggest an idea to implement in the simcore plaform
+name: <!-- Feature request -->
+about: <!-- Suggest an idea to implement in the simcore plaform -->
 
 ---
 
 **USER STORY:**
 
-A clear and concise description of how the feature works and looks like from the user's perspective.
+<!--A clear and concise description of how the feature works and looks like from the user's perspective.
 
-Ex. I want to be able the stop the running pipeline by pressing a stop button. If the pipeline is stopped, I see a info-level message confirming it in the logger, if it fails the message should be displayed in red (error). Also, all the progress bars in the nodes must be set to 0.
+Ex. I want to be able the stop the running pipeline by pressing a stop button. If the pipeline is stopped, I see a info-level message confirming it in the logger, if it fails the message should be displayed in red (error). Also, all the progress bars in the nodes must be set to 0. -->
 
 **DEFINITION OF DONE:**
 
-A clear and concise description of what the feature requires.
+<!--A clear and concise description of what the feature requires.
 
 0. Play button turns into stop button when pipeline is running
 1. Stop button turns into play button when pipeline is finished
 2. Stop button turns into play button when pipeline is successfully stopped
 3. Logger displays messages everytime the play/stop button is pressed
 4. Progress bars are set to 0 when stopping pipeline
 5. Stop button has a Python interface
+-->
 
 **EXAMPLE:**
 
-Any file/screenshot/photomontage/video/website is provided for a better understanding of the request
+<!-- Any file/screenshot/photomontage/video/website is provided for a better understanding of the request 
+
+Recommended tools for screen captures:
+	- https://getgreenshot.org/       : very complete screen capture with annotation capabilities [windows only]
+	- https://www.cockos.com/licecap/ : produces lighweight animated screen captures
+-->

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,14 +1,4 @@
-<!-- Thank you for your contribution!
-
- - Add **WIP-** prefix in title if it is still work in progress (i.e. should not be merged but is ready to start reviewing it)
- - Use [waffle.io] keywords in title/descriptions to trigger bot actions:
-   - close, closes, close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved #[issueNumber]
-   - connect to, connects to, connected to, connect, connects, connected #[issueNumber]
-   - parent of, parent to, parent #[issueNumber]
-   - child of, child to, child #[issueNumber]
-   - blocks, block, required by, needed by, dependency of #[issueNumber]
-   - blocked by, required, require, needs, need, depends on #[issueNumber]
--->
+<!--  **WIP-** prefix in title if still work in progress -->
 
 ## What do these changes do?
 
@@ -17,17 +7,20 @@
 
 ## Related issue number
 
-<!-- Are there any issues opened that will be resolved by merging this change?
+<!--
+ Use [waffle.io] keywords in title/descriptions to trigger bot actions:
 
- - add mentions of the user or team responsible for reviewing proposed changes
- - and user/team to .github/CODEOWNERS to automatically assign reviewers
+   - close, closes, close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved #[issueNumber]
+   - connect to, connects to, connected to, connect, connects, connected #[issueNumber]
 -->
 
+
 ## Checklist
 
 - [ ] I think the code is well written
 - [ ] Unit tests for the changes exist
 - [ ] Documentation reflects the changes
+- [ ] **Runs in the swarm**
 - [ ] If you design a new module, add your user to .github/CODEOWNERS
 
 [waffle.io]:https://waffle.io/marketing-assets/documents/waffleio_cheatsheet_v1.pdf?utm_source=blog&utm_medium=cheatsheet-ctabutton&utm_campaign=cheatsheet

.travis.yml

@@ -1,9 +1,11 @@
 language: python
+dist: xenial
 addons:
   apt:
     packages:
-      - "python3"
+      - python3
       - "python3-pip"
+      - docker-ce
 python:
   - "3.6"
 install:

.vscode-template/install-dev-tools.sh

@@ -0,0 +1,3 @@
+
+# uname -s == Darwin
+brew install gnu-sed

README.md

@@ -19,12 +19,28 @@ To generate a new cookiecutter template layout just type
 ```
 and answer the questions.
 
+
+## Backed project
+
+Infrastructure:
+  - [osparc-simcore]-compatible project skeleton for python services
+  - service-level makefile (type ``make help``)
+  - [folder]({{cookiecutter.project_slug}}/extra_osparc-simcore) with stub code to integrate service in [osparc-simcore] framework
+  - Multi-stage [Dockerfile]({{cookiecutter.project_slug}}/Dockerfile) with different boot modes and healt-check
+  -
+  - ...
+
+Service Features:
+  - [aiohttp] server
+  - [OpenAPI](https://www.openapis.org/) compatible RESTful API
+  - ...
+
 Then, to work in the backed project, this is the standard wordflow
 ``` console
   $ cd my project
   $ make help
 
-  # Create and activate virtual envi
+  # Create and activate virtual environment
   $ make venv
   $ source .venv/bin/activate
 
@@ -52,26 +68,6 @@ clean – cleans projects directory
 clean-force – cleans & removes also venv folder
 ```
 
-#### Features of the backed project
-  - [aiohttp] server
-  - [OpenAPI](https://www.openapis.org/) compatible RESTful API
-  -
-  - Predefined project skeleton
-  - ``.vscode-template`` are recommended settings for vscode
-  - ``{{cookiecutter.project_slug}}/extra`` contains code modules within a tree folder structure equivalent to [osparc-simcore] repo
-  - makefile
-  ```console
-  $ make help
-  simcore_service_dummy_service:0.1.0
-
-  venv – Create the virtual environment
-  install – Install project sources in "development mode"
-  uninstall – Uninstall project sources
-  test – Runs unit tests [w/ fail fast]
-  help – Display all callable targets
-  clean – Clean the project's directory
-  ```
-
 ---
 
 ## Acknoledgements
@@ -81,6 +77,6 @@ This template was built upon ideas/snippets borrowed from already existing great
 This project is licensed under the terms of the [MIT License](/LICENSE)
 
 
-[aiohptt]:https://aiohttp.readthedocs.io/en/stable/
+[aiohttp]:https://aiohttp.readthedocs.io/en/stable/
 [cookiecutter]:https://cookiecutter.readthedocs.io
 [osparc-simcore]:https://github.com/ITISFoundation/osparc-simcore

cookiecutter.json

@@ -1,18 +1,23 @@
 {
     "full_name": "Your name",
     "github_username": "Your github username",
+
     "project_name": "Name of the project",
     "project_short_description": "A short description of the project",
     "project_slug": "{{ cookiecutter.project_name.lower().replace(' ', '-') }}",
-    "dockercompose_service_name": "{{ cookiecutter.project_slug }}",
-    "dockercompose_service_api_port": "8080",
+
     "package_name": "simcore_service_{{ cookiecutter.project_slug|replace('-','_') }}",
     "distribution_name": "simcore-service-{{ cookiecutter.project_slug }}",
     "command_line_interface_bin_name": "{{ cookiecutter.distribution_name }}",
-    "simcore_install_root": ["../../", "git+https://github.com/ITISFoundation/osparc-simcore.git@master"],
-    "release_date": "{% now 'local' %}",
+
     "version": "0.1.0",
     "openapi_specs_version": "v{{ cookiecutter.version.split('.')[0] }}",
-    "sdk_package_name": "simcore_{{ cookiecutter.project_slug|replace('-','_') }}_sdk",
+
+    "dockercompose_service_name": "{{ cookiecutter.project_slug }}",
+    "dockercompose_service_api_port": "8080",
+
+    "enable_aiohttp_swagger": ["false", "true"],
+
+    "year": "{% now 'utc', '%Y' %}",
     "_extensions": ["jinja2_time.TimeExtension"]
 }

{{cookiecutter.project_slug}}/client-sdk/README.md renamed to keep/{# client-sdk #}/README.md

@@ -1,5 +1,5 @@
 #/bin/bash
-# TODO: unify scripts
+# FIXME: activate when ITIFoundation/openapi-tools:latest available
 exec {{ cookiecutter.simcore_install_root }}../scripts/openapi/openapi_codegen.sh \
     -i ../src/{{ cookiecutter.package_name }}/oas3/{{cookiecutter.openapi_specs_version}}/openapi.yaml \
     -o . \

{{cookiecutter.project_slug}}/client-sdk/codegen.sh renamed to keep/{# client-sdk #}/codegen.sh

@@ -0,0 +1,9 @@
+
+
+# TODO: docker-compose -f services/docker-compose.yml up --exit-code-from={{ cookiecutter.project_name }}  {{ cookiecutter.project_name }}
+
+# TODO: docker-compose -f services/docker-compose.yml up --detach {{ cookiecutter.project_name }}
+# test if responsive ??
+
+# ls home
+# ls /devel

{{cookiecutter.project_slug}}/client-sdk/codegen_config.json renamed to keep/{# client-sdk #}/codegen_config.json

@@ -1,14 +1,19 @@
-#pylint: disable=W0621
-# W0621:Redefining name 'here' from outer scope (line 12)
+# pylint:disable=wildcard-import
+# pylint:disable=unused-import
+# pylint:disable=unused-variable
+# pylint:disable=unused-argument
+# pylint:disable=redefined-outer-name
+
+import logging
 import os
-import sys
-import pytest
+import shutil
 import subprocess
-import logging
+import sys
 from contextlib import contextmanager
-
 from pathlib import Path
 
+import pytest
+
 logger = logging.getLogger(__name__)
 
 @pytest.fixture
@@ -38,6 +43,8 @@ def inside_dir(dirpath):
         os.chdir(old_path)
 
 
+
+
 def test_project_tree(cookies):
     result = cookies.bake(extra_context={'project_slug': 'test_project'})
     assert result.exit_code == 0
@@ -51,13 +58,28 @@ def test_project_tree(cookies):
 #    with inside_dir(str(result.project)):
 #        assert subprocess.check_call(['flake8']) == 0
 
+# TODO: use pylint via package instead of application entrypoint
 def test_run_pylint(cookies, pylintrc):
     result = cookies.bake(extra_context={'project_slug': 'pylint_compat', 'package_name': 'package_folder'})
     with inside_dir(str(result.project)):
         cmd = 'pylint --rcfile {} -v src/package_folder/'.format(pylintrc.absolute()).split()
         assert subprocess.check_call(cmd) == 0
 
 
+def test_no_tags(cookies):
+    exclude = ['.pylintrc']
+    result = cookies.bake(extra_context={
+                          'project_slug': 'myproject', 'package_name': 'package_folder'})
+    for root, dirs, files in os.walk(result.project):
+        for fname in files:
+            if fname not in exclude:
+                fpath = os.path.join(root, fname)
+                with open(fpath) as fh:
+                    for lineno, line in enumerate(fh):
+                        assert "TODO" not in line, "{}:{}".format(fpath, lineno)
+        # skips
+        dirs[:] = [n for n in dirs if not n.startswith('.')]
+
 def test_run_tests(cookies):
     result = cookies.bake(extra_context={'project_slug': 'dummy-project'})
     working_dir = str(result.project)
@@ -73,3 +95,39 @@ def test_run_tests(cookies):
             logger.info("Running '%s' ...", cmd)
             assert subprocess.check_call(cmd.split()) == 0
             logger.info("Done '%s' .", cmd)
+
+
+
+
+@pytest.mark.skip("TODO: Under development")
+def test_build_docker(cookies, tmpdir):
+    # TODO: check build target base, build, cache, prod and devel
+
+    # bakes cookie within osparc-simcore tree structure
+    result = cookies.bake(extra_context={'project_slug': 'dummy-project'})
+    working_dir = str(result.project)
+
+    tmpdir.mkdir("packages").join("dummy.py").write("import os")
+    new_working_dir = tmpdir.mkdir("services") / os.path.basename(working_dir)
+    shutil.move(working_dir, new_working_dir)
+
+    # ----
+    commands = (
+        "ls -la .",
+        "pip install pip-tools",
+        "make requirements",
+        "docker build -f Dockerfile -t dummy-project:prod --target production ../../"
+    )
+    with inside_dir(new_working_dir):
+        for cmd in commands:
+            logger.info("Running '%s' ...", cmd)
+            assert subprocess.check_call(cmd.split()) == 0
+            logger.info("Done '%s' .", cmd)
+
+
+@pytest.mark.skip("TODO: Under development")
+def test_run_docker(cookies, tmpdir):
+    # check state after boot
+    # check run permissions `simcore-service-storage --help`
+    # check load config `simcore-service-storage -c `
+    pass

{{cookiecutter.project_slug}}/_osparc-simcore/services/{{cookiecutter.project_slug}}/.gitkeep renamed to keep/{# client-sdk #}/python/.gitkeep

@@ -0,0 +1,16 @@
+:: cleanup settings
+::
+:: UNDER development 
+::
+@echo OFF
+setlocal
+
+set HERE=%~dp0
+set REPODIR=%HERE%..\
+
+set SETTINGSPATH="%REPODIR%\.vscode\settings.json"
+type %SETTINGSPATH% | python -m json.tool --sort-keys > %HERE%workspace-settings.json
+:: TODO this should update %SETTINGSPATH%
+
+set SETTINGSPATH="%APPDATA%\Code\User\settings.json"
+type %SETTINGSPATH% | python -m json.tool --sort-keys > %HERE%user-settings.json

{{cookiecutter.project_slug}}/client-sdk/sample.py renamed to keep/{# client-sdk #}/sample.py

@@ -0,0 +1,29 @@
+{
+    "atomKeymap.promptV3Features": true,
+    "editor.formatOnPaste": true,
+    "editor.insertSpaces": false,
+    "editor.multiCursorModifier": "ctrlCmd",
+    "editor.renderWhitespace": "all",
+    "editor.tabSize": 2,
+    "explorer.confirmDelete": false,
+    "files.exclude": {
+        "**/*.pyc": true,
+        "**/.DS_Store": true,
+        "**/.git": true,
+        "**/.hg": true,
+        "**/.svn": true,
+        "**/CVS": true
+    },
+    "git.enabled": true,
+    "git.path": "%ConEmuDir%\\vendor\\git-for-windows\\cmd\\git.exe",
+    "latex-workshop.view.pdf.viewer": "tab",
+    "python.linting.enabled": true,
+    "python.pythonPath": "C:\\Users\\crespo\\Miniconda3\\envs\\py3\\python.exe",
+    "python.unitTest.unittestEnabled": true,
+    "terminal.integrated.shell.windows": "cmd.exe",
+    "terminal.integrated.shellArgs.windows": [
+        "/k",
+        "%ConEmuDir%\\vendor\\init.bat"
+    ],
+    "workbench.iconTheme": "vscode-icons"
+}