garbage collector system overhaul and extras (#1724)

* guests without projects are now removed

* makes no senso for GUEST users

* garbage collection should result more descirptive

* added more notes

* added more notes

* minor refactor

* fixed error

* functions have been split into smaller self contained entities

* removed outdated description

* moved code blocks up

* removed unused function

* moved function position

* changed import name

* gc takes into consideration access rights when removing users

* fixed pylint issues

* removed unused code

* refactored code

* updated changelog

* added link in changelog

* PRs are now linked in the changelog

* marking container nvidia-smi for autoremove

* during testing aiohttp logs are displayed
- applied black reformat

* removed unencessary comments

* fixed issue which caused silent 401
- @sanderegg  there was an error but no traceback was presnet

* replaced with a more updated version

* added empty_project_data and trapped a new error

* updated dependencies

* removed unused import

* added tests for garbage collection

* fixed some dependencies

* conftest refactoring

* now creates a template db to be used between tests

* tests correctly run in integration testing

* refactored tests
implemnted T4

* fixed an issue with project ownership detection
test_t5 was implemented

* implemented t6

* implemented t7

* implemented t8

* implemented t9

* updated changelog

* removed usless test module

* @sanderegg suggestion

* @sanderegg should adds clarity to the reasoning

* @sanderegg renaming function to something more clear

* removed typo

* @sanderegg  removed redundant comments

* @sanderegg refactored fixture into a function

* fixing out more typos

* minor refactor and cleanup

* introduce tests to handle multiple user deletion
- fixed a bug
- added 2 tests

* removed uneeded comment

* used @pcrespov suggestion and refactored

* reverting depenencies for fugure upgrade @pcrespov

* moved in a more adequate palce

* changed outdated pytest fixture

Co-authored-by: Andrei Neagu <[email protected]>

Author: GitHK

CHANGELOG.md

@@ -12,12 +12,15 @@ FIXME: Compare shows single commit. SEE https://github.com/ITISFoundation/osparc
 ### Added
 
 - Started this *human-readable* CHANGELOG
-- ``migration`` service that discovers postgres service and upgrades main database (#1714)
-- Every group can register official classifiers for studies and services. Diplayed as a tree in UI (#1670, #1719, #1722)
+- ``migration`` service that discovers postgres service and upgrades main database [#1714](https://github.com/ITISFoundation/osparc-simcore/pull/1714)
+- Every group can register official classifiers for studies and services. Diplayed as a tree in UI [#1670](https://github.com/ITISFoundation/osparc-simcore/pull/1670), [#1719](https://github.com/ITISFoundation/osparc-simcore/pull/1719) , [#1722](https://github.com/ITISFoundation/osparc-simcore/pull/1722)
+- GC tests are run in isolation with a template database [#1724](https://github.com/ITISFoundation/osparc-simcore/pull/1724)
 
 ### Changed
 
-- Speedup testing by splitting webserver (#1711)
+- Speedup testing by splitting webserver [#1711](https://github.com/ITISFoundation/osparc-simcore/pull/1711)
+- Refactored garbage collector and added tests [#1724](https://github.com/ITISFoundation/osparc-simcore/pull/1724)
+- Logs are now displayed during testing [#1724](https://github.com/ITISFoundation/osparc-simcore/pull/1724)
 
 <!-- ### Deprecated -->
 <!-- ### Removed    -->
@@ -28,45 +31,45 @@ FIXME: Compare shows single commit. SEE https://github.com/ITISFoundation/osparc
 ## [0.0.25] - 2020-08-04
 
 ### Added
-- add traefik endpoint to api-gateway (#1555)
-- Shared project concurrency (frontend) (#1591)
-- Homogenize studies and services (#1569)
+- add traefik endpoint to api-gateway [#1555](https://github.com/ITISFoundation/osparc-simcore/pull/1555)
+- Shared project concurrency (frontend) [#1591](https://github.com/ITISFoundation/osparc-simcore/pull/1591) 
+- Homogenize studies and services [#1569](https://github.com/ITISFoundation/osparc-simcore/pull/1569)
 - UI Fine grained access - project locking and notification
-- Adds support for GPU scheduling of computational services (#1553)
+- Adds support for GPU scheduling of computational services [#1553](https://github.com/ITISFoundation/osparc-simcore/pull/1553)
 
 ### Changed
-- UI/UX improvements (#1657)
-- Improving storage performance (#1659)
-- Theming (#1656)
-- Reduce cardinality of metrics (#1593)
+- UI/UX improvements [#1657](https://github.com/ITISFoundation/osparc-simcore/pull/1657) 
+- Improving storage performance [#1659](https://github.com/ITISFoundation/osparc-simcore/pull/1659)
+- Theming [#1656](https://github.com/ITISFoundation/osparc-simcore/pull/1656)
+- Reduce cardinality of metrics [#1593](https://github.com/ITISFoundation/osparc-simcore/pull/1593) 
 
 ### Fixed
-- Platform stability:  (#1645)
-- Fix, improves and re-activate e2e CI testing (#1594, #1620, #1631, #1600)
-- Fixes defaults (#1640)
-- Upgrade storage service (#1585, #1586)
-- UPgrade catalog service (#1582)
-- Fixes on publish studies handling (#1632)
-- Invalidate cache before starting a study (#1602)
-- Some enhancements and bug fixes (#1608)
-- filter studies by name before deleting them (#1629)
-- Bugfix/apiserver does not need sslheaders (#1564)
-- fix testing if node has gpu support (#1604)
-- /study fails 500 (#1570, #1572)
-- fix codecov reports (#1568)
+- Platform stability: [#1645](https://github.com/ITISFoundation/osparc-simcore/pull/1645)
+- Fix, improves and re-activate e2e CI testing [#1594](https://github.com/ITISFoundation/osparc-simcore/pull/1594), [#1620](https://github.com/ITISFoundation/osparc-simcore/pull/1620), [#1631](https://github.com/ITISFoundation/osparc-simcore/pull/1631), [#1600](https://github.com/ITISFoundation/osparc-simcore/pull/1600)
+- Fixes defaults [#1640](https://github.com/ITISFoundation/osparc-simcore/pull/1640)
+- Upgrade storage service [#1585](https://github.com/ITISFoundation/osparc-simcore/pull/1585), [#1586](https://github.com/ITISFoundation/osparc-simcore/pull/1586)
+- UPgrade catalog service [#1582](https://github.com/ITISFoundation/osparc-simcore/pull/1582) 
+- Fixes on publish studies handling [#1632](https://github.com/ITISFoundation/osparc-simcore/pull/1632)
+- Invalidate cache before starting a study [#1602](https://github.com/ITISFoundation/osparc-simcore/pull/1602)
+- Some enhancements and bug fixes [#1608](https://github.com/ITISFoundation/osparc-simcore/pull/1608)
+- filter studies by name before deleting them [#1629](https://github.com/ITISFoundation/osparc-simcore/pull/1629)
+- Bugfix/apiserver does not need sslheaders [#1564](https://github.com/ITISFoundation/osparc-simcore/pull/1564)
+- fix testing if node has gpu support [#1604](https://github.com/ITISFoundation/osparc-simcore/pull/1604)
+- /study fails 500 [#1570](https://github.com/ITISFoundation/osparc-simcore/pull/1570), [#1572](https://github.com/ITISFoundation/osparc-simcore/pull/1572)
+- fix codecov reports [#1568](https://github.com/ITISFoundation/osparc-simcore/pull/1568)
 
 ### Security
-- Bump yarl from 1.4.2 to 1.5.1 in /packages/postgres-database (#1665)
-- Bump ujson from 3.0.0 to 3.1.0 in /packages/service-library (#1664)
-- Bump pytest-docker from 0.7.2 to 0.8.0 in /packages/service-library (#1647)
-- Bump aiozipkin from 0.6.0 to 0.7.0 in /packages/service-library (#1642)
-- Bump lodash from 4.17.15 to 4.17.19 (#1639)
-- Maintenance/upgrades test tools (#1628)
-- Bugfix/concurent opening projects (#1598)
-- Bugfix/allow reading groups anonymous user (#1615)
-- Bump docker from 4.2.1 to 4.2.2 in /packages/postgres-database (#1605)
-- Bump faker from 4.1.0 to 4.1.1 in /packages/postgres-database (#1573)
-- Maintenance/upgrades and tooling (#1546)
+- Bump yarl from 1.4.2 to 1.5.1 in /packages/postgres-database [#1665](https://github.com/ITISFoundation/osparc-simcore/pull/1665)
+- Bump ujson from 3.0.0 to 3.1.0 in /packages/service-library [#1664](https://github.com/ITISFoundation/osparc-simcore/pull/1664)
+- Bump pytest-docker from 0.7.2 to 0.8.0 in /packages/service-library [#1647](https://github.com/ITISFoundation/osparc-simcore/pull/1647)
+- Bump aiozipkin from 0.6.0 to 0.7.0 in /packages/service-library [#1642](https://github.com/ITISFoundation/osparc-simcore/pull/1642)
+- Bump lodash from 4.17.15 to 4.17.19 [#1639](https://github.com/ITISFoundation/osparc-simcore/pull/1639)
+- Maintenance/upgrades test tools [#1628](https://github.com/ITISFoundation/osparc-simcore/pull/1628)
+- Bugfix/concurent opening projects [#1598](https://github.com/ITISFoundation/osparc-simcore/pull/1598)
+- Bugfix/allow reading groups anonymous user [#1615](https://github.com/ITISFoundation/osparc-simcore/pull/1615)
+- Bump docker from 4.2.1 to 4.2.2 in /packages/postgres-database [#1605](https://github.com/ITISFoundation/osparc-simcore/pull/1605)
+- Bump faker from 4.1.0 to 4.1.1 in /packages/postgres-database [#1573](https://github.com/ITISFoundation/osparc-simcore/pull/1573)
+- Maintenance/upgrades and tooling [#1546](https://github.com/ITISFoundation/osparc-simcore/pull/1546)
 
 
 ---

packages/postgres-database/src/simcore_postgres_database/cli.py

@@ -37,7 +37,11 @@
 discovered_cache = os.path.expanduser("~/.simcore_postgres_database_cache.json")
 
 log = logging.getLogger("root")
-fileConfig(default_ini)
+
+if __name__ == "__main__":
+    # swallows up all log messages from tests
+    # only enable it during cli invocation
+    fileConfig(default_ini)
 
 
 def safe(if_fails_return=False):

packages/postgres-database/src/simcore_postgres_database/migration/env.py

@@ -1,4 +1,3 @@
-
 from logging.config import fileConfig
 
 from alembic import context
@@ -12,7 +11,11 @@
 
 # Interpret the config file for Python logging.
 # This line sets up loggers basically.
-fileConfig(config.config_file_name)
+
+if __name__ == "__main__":
+    # swallows up all log messages from tests
+    # only enable it during cli invocation
+    fileConfig(config.config_file_name)
 
 # add your model's MetaData object here
 # for 'autogenerate' support
@@ -38,9 +41,7 @@ def run_migrations_offline():
     """
     # pylint: disable=no-member
     url = config.get_main_option("sqlalchemy.url")
-    context.configure(
-        url=url, target_metadata=target_metadata, literal_binds=True
-    )
+    context.configure(url=url, target_metadata=target_metadata, literal_binds=True)
 
     with context.begin_transaction():
         context.run_migrations()
@@ -61,9 +62,7 @@ def run_migrations_online():
     )
 
     with connectable.connect() as connection:
-        context.configure(
-            connection=connection, target_metadata=target_metadata
-        )
+        context.configure(connection=connection, target_metadata=target_metadata)
 
         with context.begin_transaction():
             context.run_migrations()

packages/pytest-simcore/src/pytest_simcore/helpers/utils_projects.py

@@ -10,6 +10,7 @@
 import re
 import uuid as uuidlib
 from typing import Dict
+from simcore_service_webserver.utils import now_str
 
 from aiohttp import web
 
@@ -32,6 +33,19 @@
 ]
 
 
+def empty_project_data():
+    return {
+        "uuid": f"project-{uuidlib.uuid4()}",
+        "name": "Empty name",
+        "description": "some description of an empty project",
+        "prjOwner": "I'm the empty project owner, hi!",
+        "creationDate": now_str(),
+        "lastChangeDate": now_str(),
+        "thumbnail": "",
+        "workbench": {},
+    }
+
+
 def load_data(name):
     with resources.stream(name) as fp:
         return json.load(fp)
@@ -62,7 +76,7 @@ async def create_project(
     try:
         uuidlib.UUID(project_data["uuid"])
         assert new_project["uuid"] == project_data["uuid"]
-    except ValueError:
+    except (ValueError, AssertionError):
         # in that case the uuid gets replaced
         assert new_project["uuid"] != project_data["uuid"]
         project_data["uuid"] = new_project["uuid"]
@@ -90,7 +104,7 @@ def __init__(
         clear_all=True,
         user_id=None,
         *,
-        force_uuid=False
+        force_uuid=False,
     ):
         self.params = params
         self.user_id = user_id

packages/pytest-simcore/src/pytest_simcore/postgres_service.py

@@ -2,22 +2,134 @@
 # pylint:disable=unused-argument
 # pylint:disable=redefined-outer-name
 
-import os
-from typing import Dict
 import logging
+import asyncio
+import os
+from typing import Dict, List
 
 import pytest
+import simcore_postgres_database.cli as pg_cli
 import sqlalchemy as sa
 import tenacity
-from sqlalchemy.orm import sessionmaker
-
-import simcore_postgres_database.cli as pg_cli
 from simcore_postgres_database.models.base import metadata
+from sqlalchemy.orm import sessionmaker
 
 from .helpers.utils_docker import get_service_published_port
 
 log = logging.getLogger(__name__)
 
+TEMPLATE_DB_TO_RESTORE = "template_simcore_db"
+
+
+def execute_queries(
+    postgres_engine: sa.engine.Engine,
+    sql_statements: List[str],
+    ignore_errors: bool = False,
+) -> None:
+    """runs the queries in the list in order"""
+    with postgres_engine.connect() as con:
+        for statement in sql_statements:
+            try:
+                con.execution_options(autocommit=True).execute(statement)
+            except Exception as e:  # pylint: disable=broad-except
+                # when running tests initially the TEMPLATE_DB_TO_RESTORE dose not exist and will cause an error
+                # which can safely be ignored. The debug message is here to catch future errors which and
+                # avoid time wasting
+                log.debug("SQL error which can be ignored %s", str(e))
+
+
+def create_template_db(postgres_dsn: Dict, postgres_engine: sa.engine.Engine) -> None:
+    # create a template db, the removal is necessary to allow for the usage of --keep-docker-up
+    queries = [
+        # disconnect existing users
+        f"""
+        SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity 
+        WHERE pg_stat_activity.datname = '{postgres_dsn["database"]}' AND pid <> pg_backend_pid();
+        """,
+        # drop template database
+        f"ALTER DATABASE {TEMPLATE_DB_TO_RESTORE} is_template false;",
+        f"DROP DATABASE {TEMPLATE_DB_TO_RESTORE};",
+        # create template database
+        """
+        CREATE DATABASE {template_db} WITH TEMPLATE {original_db} OWNER {db_user};
+        """.format(
+            template_db=TEMPLATE_DB_TO_RESTORE,
+            original_db=postgres_dsn["database"],
+            db_user=postgres_dsn["user"],
+        ),
+    ]
+    execute_queries(postgres_engine, queries, ignore_errors=True)
+
+
+def drop_template_db(postgres_engine: sa.engine.Engine) -> None:
+    # remove the template db
+    queries = [
+        # drop template database
+        f"ALTER DATABASE {TEMPLATE_DB_TO_RESTORE} is_template false;",
+        f"DROP DATABASE {TEMPLATE_DB_TO_RESTORE};",
+    ]
+    execute_queries(postgres_engine, queries)
+
+
+@pytest.fixture(scope="module")
+def loop(request):
+    loop = asyncio.get_event_loop_policy().new_event_loop()
+    yield loop
+    loop.close()
+
+
+@pytest.fixture(scope="module")
+def postgres_with_template_db(
+    postgres_db: sa.engine.Engine, postgres_dsn: Dict, postgres_engine: sa.engine.Engine
+) -> sa.engine.Engine:
+    create_template_db(postgres_dsn, postgres_engine)
+    yield postgres_engine
+    drop_template_db(postgres_engine)
+
+
+@pytest.fixture
+def drop_db_engine(postgres_dsn: Dict) -> sa.engine.Engine:
+    postgres_dsn_copy = postgres_dsn.copy()  # make a copy to change these parameters
+    postgres_dsn_copy["database"] = "postgres"
+    dsn = "postgresql://{user}:{password}@{host}:{port}/{database}".format(
+        **postgres_dsn_copy
+    )
+    return sa.create_engine(dsn, isolation_level="AUTOCOMMIT")
+
+
+@pytest.fixture
+def database_from_template_before_each_function(
+    postgres_dsn: Dict, drop_db_engine: sa.engine.Engine, postgres_db
+) -> None:
+    """
+    Will recrate the db before running each test.
+
+    **Note: must be implemented in the module where the the
+    `postgres_with_template_db` is used and mark autouse=True**
+
+    It is possible to drop the application database by ussing another one like
+    the posgtres database. The db will be recrated from the previously created template
+
+    The postgres_db fixture is required for the template database to be created.
+    """
+
+    queries = [
+        # terminate existing connections to the database
+        f"""
+        SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity 
+        WHERE pg_stat_activity.datname = '{postgres_dsn["database"]}';
+        """,
+        # drop database
+        f"DROP DATABASE {postgres_dsn['database']};",
+        # create from template database
+        f"CREATE DATABASE {postgres_dsn['database']} TEMPLATE template_simcore_db;",
+    ]
+
+    execute_queries(drop_db_engine, queries)
+
+    yield
+    # do nothing on teadown
+
 
 @pytest.fixture(scope="module")
 def postgres_dsn(docker_stack: Dict, devel_environ: Dict) -> Dict[str, str]: