Skip to content

♻️Storage: Code refactoring prior to changes (🚨🚨) #7088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 181 commits into from
Feb 10, 2025

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Jan 24, 2025

What do these changes do?

Very noisy PR that refactors storage service from aiohttp- to fastapi-based service.

  • aiohttp --> fastapi, all endpoints were modified, also the long running tasks and the client to the long running tasks that needed a fastapi compatible version
  • aiopg --> asyncpg, this brought a lot of issues regarding the fact that asyncpg does not seemlessly convert between strings and postgres types (along these lines, had to revert to use non time aware types until the database is updated)
  • removed the API entrypoint for syncing the table and S3 since it is completely unused
  • cleanup, refactoring and re-structuring alongside our other fastapi-based services

Important note: @bisgaard-itis @GitHK @mrnicegyu11 @odeimaiz @matusdrobuliak66 storage cannot be run as several replicas as I could not find a way to get the IP of the server via call to request during uploads of files. That means this will need a subsequent PR that will keep the state of the upload in the DB or call into AWS S3 until the completion of the S3 multipart is done. There will be a follow up on that in 1 week.

Related issue/s

How to test

Dev-ops checklist

@sanderegg sanderegg added a:storage issue related to storage service t:maintenance Some planned maintenance work labels Jan 24, 2025
@sanderegg sanderegg added this to the Singularity milestone Jan 24, 2025
@sanderegg sanderegg self-assigned this Jan 24, 2025
Copy link

codecov bot commented Jan 24, 2025

Codecov Report

Attention: Patch coverage is 84.58150% with 140 lines in your changes missing coverage. Please review.

Project coverage is 87.76%. Comparing base (6fd6729) to head (36bd804).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7088      +/-   ##
==========================================
+ Coverage   87.27%   87.76%   +0.48%     
==========================================
  Files        1641     1628      -13     
  Lines       64222    63226     -996     
  Branches     1179     1093      -86     
==========================================
- Hits        56051    55488     -563     
+ Misses       7860     7426     -434     
- Partials      311      312       +1     
Flag Coverage Δ
integrationtests 65.36% <30.00%> (+0.64%) ⬆️
unittests 86.03% <84.47%> (+0.47%) ⬆️
Components Coverage Δ
api 76.84% <ø> (ø)
pkg_aws_library 94.02% <100.00%> (ø)
pkg_dask_task_models_library 97.09% <ø> (ø)
pkg_models_library 91.54% <100.00%> (+0.02%) ⬆️
pkg_notifications_library 84.57% <ø> (ø)
pkg_postgres_database 88.21% <ø> (ø)
pkg_service_integration 70.03% <ø> (ø)
pkg_service_library 72.96% <43.62%> (-1.18%) ⬇️
pkg_settings_library 90.71% <ø> (ø)
pkg_simcore_sdk 85.51% <100.00%> (+<0.01%) ⬆️
agent 96.46% <ø> (ø)
api_server 90.55% <ø> (ø)
autoscaling 96.08% <ø> (ø)
catalog 91.71% <100.00%> (+0.01%) ⬆️
clusters_keeper 99.24% <ø> (ø)
dask_sidecar 91.26% <ø> (ø)
datcore_adapter 93.19% <ø> (ø)
director 76.59% <100.00%> (-0.43%) ⬇️
director_v2 91.30% <ø> (-0.08%) ⬇️
dynamic_scheduler 97.17% <ø> (ø)
dynamic_sidecar 89.75% <ø> (ø)
efs_guardian 90.25% <ø> (ø)
invitations 93.28% <ø> (ø)
osparc_gateway_server ∅ <ø> (∅)
payments 92.66% <ø> (ø)
resource_usage_tracker 89.01% <ø> (-0.11%) ⬇️
storage 86.22% <91.92%> (-3.35%) ⬇️
webclient ∅ <ø> (∅)
webserver 87.09% <97.87%> (+2.34%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6fd6729...36bd804. Read the comment docs.

@sanderegg sanderegg force-pushed the storage/refactoring branch 6 times, most recently from 2eaa872 to dd968f9 Compare January 29, 2025 21:59
@sanderegg sanderegg force-pushed the storage/refactoring branch 2 times, most recently from 65be5bc to 2c7fcea Compare January 30, 2025 11:19
@sanderegg sanderegg marked this pull request as ready for review January 30, 2025 11:19
@sanderegg sanderegg changed the title ♻️Storage: Code refactoring prior to changes ♻️Storage: Code refactoring prior to changes (🚨🚨) Jan 30, 2025
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, just some more comments

Copy link

@sanderegg sanderegg added the 🤖-automerge marks PR as ready to be merged for Mergify label Feb 10, 2025
@GitHK
Copy link
Contributor

GitHK commented Feb 10, 2025

@Mergifyio queue

Copy link
Contributor

mergify bot commented Feb 10, 2025

queu

❌ Sorry but I didn't understand the command. Please consult the commands documentation 📚.

Copy link
Contributor

mergify bot commented Feb 10, 2025

queue

🛑 The pull request has been merged manually

The pull request has been merged manually at 3bb98a6

@sanderegg sanderegg merged commit 3bb98a6 into ITISFoundation:master Feb 10, 2025
101 of 106 checks passed
@sanderegg sanderegg deleted the storage/refactoring branch February 10, 2025 14:24
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Mar 6, 2025
63 tasks
mrnicegyu11 pushed a commit to mrnicegyu11/osparc-simcore that referenced this pull request Mar 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🤖-automerge marks PR as ready to be merged for Mergify a:storage issue related to storage service t:maintenance Some planned maintenance work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix openapi specs generation of storage maintenance: long running tasks for FastAPI-based app needs upgrade
7 participants