✨Computation of path size: WebAPI + RPC to storage through Asyncjobs

[sanderegg]

What do these changes do?

This PR brings a new asynchronous entrypoint in the webserver API for computing path sizes:

The call in the webserver is transmitted to the storage microservice via RPC, then to its storage-worker micro-service via Celery. The status/result of the computation is retrieved via GET /storage/async-jobs/{job_id}/status and GET /storage/async-jobs/{job_id}/result respectively.

graph LR
    subgraph Web
        Client["Web Client"] -->|POST /paths/path:size| Server["Web Server"];
    end
    subgraph Storage
        Server -->|RPC: Compute Path Size| StorageRPC["Storage RPC Handler"];
        StorageRPC -->|Celery Task: compute_path_size| StorageWorker["Storage Worker"];
        StorageRPC -->|Return AsyncJobUUID| Client;
    end

Loading

How to use

• POST /v0/storage/locations/0/paths/50d72a1c-f816-11ef-8af3-0242ac140610:size --> returns a jobId
• GET /v0/storage/async-jobs/ce05e55f-c57a-45f0-9b12-d3dc79de0be6/status returns the job status
• GET /v0/storage/async-jobs/ce05e55f-c57a-45f0-9b12-d3dc79de0be6/result returns the job result --> byte size of the path

Bonus:

• some refactoring: created _worker_tasks api subfolder to contain all celery tasks definition
• refactored define_task to handle automatically async methods
• Allow Path to be passed through Celery tasks (NOTE: Celery 5.5. will be compatible with Pydantic types!)
• added watchdog for devel mode

Related issue/s

• requires ✨Storage: provide entrypoint to compute path size (⚠️ devops) #7286

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 67.64706% with 44 lines in your changes missing coverage. Please review.

Project coverage is 87.85%. Comparing base (189821d) to head (08f0994).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7315      +/-   ##
==========================================
+ Coverage   78.67%   87.85%   +9.18%     
==========================================
  Files        1179     1526     +347     
  Lines       47798    61533   +13735     
  Branches      552      891     +339     
==========================================
+ Hits        37605    54062   +16457     
+ Misses      10038     7212    -2826     
- Partials      155      259     +104     

Flag	Coverage Δ
integrationtests	65.35% <50.00%> (+0.02%)	⬆️
unittests	86.23% <67.64%> (+10.18%)	⬆️

Components	Coverage Δ
api	76.84% <ø> (ø)
pkg_aws_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	91.74% <100.00%> (+<0.01%)	⬆️
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	70.03% <ø> (ø)
pkg_service_library	72.32% <0.00%> (∅)
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	85.46% <ø> (ø)
agent	96.46% <ø> (ø)
api_server	90.68% <ø> (ø)
autoscaling	96.08% <ø> (∅)
catalog	92.14% <ø> (ø)
clusters_keeper	99.24% <ø> (ø)
dask_sidecar	91.25% <ø> (∅)
datcore_adapter	98.11% <ø> (ø)
director	76.59% <ø> (∅)
director_v2	91.30% <ø> (+12.75%)	⬆️
dynamic_scheduler	97.33% <ø> (∅)
dynamic_sidecar	90.09% <ø> (+0.98%)	⬆️
efs_guardian	89.79% <ø> (∅)
invitations	93.28% <ø> (ø)
osparc_gateway_server	∅ <ø> (∅)
payments	92.66% <ø> (ø)
resource_usage_tracker	89.18% <ø> (∅)
storage	84.22% <72.47%> (∅)
webclient	∅ <ø> (∅)
webserver	88.18% <100.00%> (+21.53%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 189821d...08f0994. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[odeimaiz]

Merci 👍

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud