🐛S3: when copying files there is no callback if the multipart threshold is not reached

[sanderegg]

What do these changes do?

#6272 did not completely fix the issue of the progress while duplicating studies or instantiating templates:

• we use boto3 which officially wraps the AWS API for python,
• the copy() function of boto3 auto-switches between copy_object and multipart copying based on a threshold that was overriden by us at a value of 5GB instead of the default value of 8MB,
• this was done as copy_object seemed much faster and seemed to not download/upload data contrary to the multipart counterpart,
• the problem here is that copy_object does not provide any kind of feedback on the copy progress,
• the copy callback for progress would only be called with files larger than 5GB, which is not ideal.

This PR therefore adds:

1. explicit call on the progress callback after the copy to ensure that at least the final feedback comes in,

• in the case of folder copying, the size of each files is known already,
• in the case of single file copying, an additional call to S3 API is done to get the size of the object.

2. reduced the multipart threshold to 100MB for now (@pcrespov I would need here your measurements to see if this is reasonable - if 5GB is copied in 2 seconds or a reasonable time, then we can revert that part)

Related issue/s

• Bug: Copying/instantiating template progress broken on prod #1604  #6209
• Progress inside the progress bar seems invisible when duplicating a study osparc-issues#1681

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 88.8%. Comparing base (cafbf96) to head (24a75ab).
Report is 505 commits behind head on master.

Files with missing lines	Patch %	Lines
packages/aws-library/src/aws_library/s3/_client.py	80.0%	0 Missing and 1 partial ⚠️
...es/storage/src/simcore_service_storage/s3_utils.py	75.0%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #6305      +/-   ##
=========================================
+ Coverage    84.5%   88.8%    +4.2%     
=========================================
  Files          10    1056    +1046     
  Lines         214   47103   +46889     
  Branches       25     408     +383     
=========================================
+ Hits          181   41843   +41662     
- Misses         23    5189    +5166     
- Partials       10      71      +61     

Flag	Coverage Δ
integrationtests	64.6% <ø> (?)
unittests	86.2% <80.0%> (+1.6%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...kages/aws-library/src/aws_library/s3/_constants.py	100.0% <100.0%> (ø)
packages/aws-library/src/aws_library/s3/_client.py	95.3% <80.0%> (ø)
...es/storage/src/simcore_service_storage/s3_utils.py	86.1% <75.0%> (ø)

... and 1050 files with indirect coverage changes

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[matusdrobuliak66]

Thanks!

[odeimaiz]

Thanks 🥇!