Skip to content

garbage collector system overhaul and extras #1724

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 68 commits into from
Sep 15, 2020
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
fcd3ca9
guests without projects are now removed
Aug 20, 2020
1efac17
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Aug 20, 2020
0afaefa
makes no senso for GUEST users
Aug 20, 2020
4620bd8
Merge remote-tracking branch 'upstream/master' into gc-remove-guests-…
Aug 21, 2020
ae7e470
garbage collection should result more descirptive
Aug 21, 2020
d5e33f8
added more notes
Aug 21, 2020
3a2f83a
added more notes
Aug 21, 2020
9f15f2c
minor refactor
Aug 21, 2020
6f2aaf5
fixed error
Aug 21, 2020
7dfb928
functions have been split into smaller self contained entities
Aug 21, 2020
dbbbb48
removed outdated description
Aug 21, 2020
7a16518
moved code blocks up
Aug 21, 2020
bba42b6
removed unused function
Aug 21, 2020
53415cb
moved function position
Aug 21, 2020
076f238
Merge remote-tracking branch 'upstream/master' into gc-remove-guests-…
Aug 31, 2020
67be62d
changed import name
Aug 31, 2020
57b5092
gc takes into consideration access rights when removing users
Sep 1, 2020
575e036
fixed pylint issues
Sep 1, 2020
519646b
removed unused code
Sep 1, 2020
59a457f
refactored code
Sep 2, 2020
1259919
updated changelog
Sep 2, 2020
2101786
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Sep 2, 2020
e47fb02
added link in changelog
Sep 2, 2020
a3bfc58
PRs are now linked in the changelog
Sep 2, 2020
55ce9e7
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Sep 2, 2020
685c5f2
marking container nvidia-smi for autoremove
Sep 3, 2020
3e7578a
during testing aiohttp logs are displayed
Sep 4, 2020
6e78b49
removed unencessary comments
Sep 7, 2020
caa3f6b
fixed issue which caused silent 401
Sep 7, 2020
3e748ea
replaced with a more updated version
Sep 8, 2020
ce07201
added empty_project_data and trapped a new error
Sep 8, 2020
5eeeb76
updated dependencies
Sep 8, 2020
d635e70
removed unused import
Sep 8, 2020
8660d17
added tests for garbage collection
Sep 8, 2020
e94a793
fixed some dependencies
Sep 8, 2020
8fa678c
conftest refactoring
Sep 8, 2020
a3e1b30
now creates a template db to be used between tests
Sep 8, 2020
3151e22
tests correctly run in integration testing
Sep 9, 2020
717e2a9
refactored tests
Sep 9, 2020
d4e5179
fixed an issue with project ownership detection
Sep 9, 2020
a9d8124
implemented t6
Sep 9, 2020
673a7ff
implemented t7
Sep 9, 2020
8d80661
implemented t8
Sep 9, 2020
6f15ca1
implemented t9
Sep 9, 2020
6764f59
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Sep 9, 2020
5a086e4
updated changelog
Sep 9, 2020
36bbae6
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Sep 9, 2020
49b8a1c
removed usless test module
Sep 9, 2020
d33f963
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Sep 10, 2020
1d9b59a
@sanderegg suggestion
Sep 10, 2020
081b052
@sanderegg should adds clarity to the reasoning
Sep 10, 2020
cfe17a2
@sanderegg renaming function to something more clear
Sep 10, 2020
8d7889a
removed typo
Sep 10, 2020
acb935d
@sanderegg removed redundant comments
Sep 10, 2020
1a77741
@sanderegg refactored fixture into a function
Sep 10, 2020
07793aa
fixing out more typos
Sep 10, 2020
cb03419
minor refactor and cleanup
Sep 10, 2020
3853210
introduce tests to handle multiple user deletion
Sep 10, 2020
28c1d9d
removed uneeded comment
Sep 10, 2020
345e41e
used @pcrespov suggestion and refactored
Sep 11, 2020
f070c1f
reverting depenencies for fugure upgrade @pcrespov
Sep 11, 2020
cca0789
moved in a more adequate palce
Sep 11, 2020
fc10058
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Sep 11, 2020
1e2007a
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Sep 11, 2020
e7458e4
changed outdated pytest fixture
Sep 14, 2020
59686c6
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Sep 15, 2020
5bb56b9
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Sep 15, 2020
7d5a2a3
Merge branch 'master' into gc-remove-guests-with-no-projects
GitHK Sep 15, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -132,9 +132,9 @@ async def remove_users_manually_marked_as_guests(
continue
logger.info("Will try to remove resources for guest '%s'", guest_user_id)
# get all projects for this user and then remove with remove_resources_if_guest_user
user_project_uuids = await app[APP_PROJECT_DBAPI].list_all_projects_by_uuid_for_user(
user_id=guest_user_id
)
user_project_uuids = await app[
APP_PROJECT_DBAPI
].list_all_projects_by_uuid_for_user(user_id=guest_user_id)
logger.info(
"Project uuids, to clean, for user '%s': '%s'",
guest_user_id,
Expand All @@ -146,6 +146,11 @@ async def remove_users_manually_marked_as_guests(
app=app, project_uuid=project_uuid, user_id=guest_user_id,
)

# if there are not projects, just remove the user
await remove_resources_if_guest_user(
app=app, project_uuid=None, user_id=guest_user_id,
)


async def remove_orphaned_services(
registry: RedisResourceRegistry, app: web.Application
Expand Down Expand Up @@ -203,14 +208,14 @@ async def remove_resources_if_guest_user(
logger.debug("User is not GUEST, skipping removal of its project resources")
return

logger.debug(
"Removing project '%s' from the database", project_uuid,
)

try:
await delete_project_from_db(app, project_uuid, user_id)
except ProjectNotFoundError:
logging.warning("Project '%s' not found, skipping removal", project_uuid)
if project_uuid is not None:
try:
logger.debug(
"Removing project '%s' from the database", project_uuid,
)
await delete_project_from_db(app, project_uuid, user_id)
except ProjectNotFoundError:
logging.warning("Project '%s' not found, skipping removal", project_uuid)

# when manually changing a user to GUEST, it might happen that it has more then one project
try:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1451,6 +1451,7 @@ async def test_open_shared_project_at_same_time(
user_role: UserRole,
expected: ExpectedResponse,
aiohttp_client,
disable_gc_manual_guest_users,
):
NUMBER_OF_ADDITIONAL_CLIENTS = 20
# log client 1
Expand Down