✨ Add exporter code to storage

[GitHK]

What do these changes do?

• Add functionality in the simcore_s3_dsm that allows to export an archive, given a list of S3 object_keys.
• Using ZIP_64 protocol instead of ZIP_32 which did not support 4GB+ files.
• Expose functionality to the frontend via the celery job endpoint.

Resolved a few issues with celery tasks:

• when registering a task some options can now be specified
• number of retry attempts (default 3)
• wait period between retries (default 5 seconds)
• timeout of a task (default 60 seconds)
• ids no longer change between retry attempts
• the error raised in the worker is now raised in the context of the caller
• rewrote test_async_jobs.py without using mocks

Limitations (mostly for @odeimaiz):

• any selection inside ´UUID/...´ is accepted, either being a folder or a single file eg: UUID or UUID/ or UUID/sub/path/to/file

Related issue/s

• UI/UX for Data Handling (Feb.) osparc-issues#1635
• closes Implement functionality for fetching files/folders from s3, zipping them and uploading the zip to s3 #7187

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 92.20183% with 17 lines in your changes missing coverage. Please review.

Project coverage is 87.30%. Comparing base (70409d2) to head (be9b1e3).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7218      +/-   ##
==========================================
- Coverage   87.36%   87.30%   -0.07%     
==========================================
  Files        1742     1592     -150     
  Lines       67228    64290    -2938     
  Branches     1140      953     -187     
==========================================
- Hits        58736    56130    -2606     
+ Misses       8171     7890     -281     
+ Partials      321      270      -51     

Flag	Coverage Δ
integrationtests	65.10% <61.53%> (-0.12%)	⬇️
unittests	86.44% <92.20%> (-0.11%)	⬇️

Components	Coverage Δ
api	76.84% <ø> (ø)
pkg_aws_library	93.91% <100.00%> (+<0.01%)	⬆️
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	91.94% <100.00%> (+<0.01%)	⬆️
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	70.03% <ø> (ø)
pkg_service_library	72.24% <33.33%> (+0.01%)	⬆️
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	85.40% <ø> (ø)
agent	96.46% <ø> (ø)
api_server	90.02% <ø> (ø)
autoscaling	96.08% <ø> (ø)
catalog	91.82% <ø> (+0.11%)	⬆️
clusters_keeper	99.24% <ø> (ø)
dask_sidecar	91.29% <ø> (ø)
datcore_adapter	98.12% <ø> (ø)
director	76.78% <ø> (ø)
director_v2	91.25% <ø> (-0.05%)	⬇️
dynamic_scheduler	97.35% <ø> (ø)
dynamic_sidecar	90.11% <ø> (ø)
efs_guardian	89.79% <ø> (ø)
invitations	93.28% <ø> (ø)
payments	92.66% <ø> (ø)
resource_usage_tracker	89.12% <ø> (ø)
storage	87.65% <95.80%> (+0.47%)	⬆️
webclient	∅ <ø> (∅)
webserver	85.88% <92.30%> (-0.01%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70409d2...be9b1e3. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sanderegg]

thanks for the changes!

[GitHK]

@Mergifyio queue

[mergify]

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at e162382

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud