Feature export/import study (backend)

[GitHK]

What do these changes do?

Adds Import/export ability for studies.

What it brings:

• max upload 10 GB (env var to change size)
• uses library for parallel downloads from storage (maybe needs better alternative)
• supports multiple import/export versions out of the box (easy to extend the format)
• Pydantic models to validate export/import contents

Missing to do in future PRs:

• wrapper for K-Core format (easily obtainable by extending what is already used)
  • Because the K-Core format is more focused on full experimental studies, all the required files (4) make no sense to have, we should establish a base version which is the same for the export format
  • the idea is to adapt to their desired directory structure without changing the current export format
• upload and download to K-Core platform
• access rights for importing and exporting (permissions are already in place for this)
  
• to consider in the future: locking the study while the export is in progress

Bonuses:

• Fixed some Pydantic models which were not behaving as expected (might break some tests)

Related issue number

ITISFoundation/osparc-issues#325

How to test

Step 1: setup tools

• Use postman and import the following profile [imp-exp.postman_collection.json.zip]
  (https://github.com/ITISFoundation/osparc-simcore/files/5738904/imp-exp.postman_collection.json.zip) NOTE: you need to unzip it
• Recover the cookie from the browser's debugger, usually found in the headers and replace it in both requests

Step 2: exporting

• Fetch the project_uuid of a project you would like to export and replace it in the URL of the export request
• in the export make sure to select Send and Download (see below image). You will be asked to save a file, do that.
  

Step 3: importing

• without changing the file's name select it as a file to upload in the import
• if successful refresh your dashboard, open the project and confirm that it's working as the previous one.

Debugging import/export project data

• From admirer export the original and imported project in csv format, you should get a file projects.csv (see setup below)
  

• Use the following script to transform projects.csv to json files which are simpler to compare csv_to_json.py.zip

Checklist

• Did you change any service's API? Then make sure to bundle document and upgrade version (make openapi-specs, git commit ... and then make version-*)
• Unit tests for the changes exist
• Runs in the swarm
• Documentation reflects the changes
• New module? Add your github username to .github/CODEOWNERS

[codecov]

Codecov Report

Merging #2053 (761a7af) into master (9e0d6df) will increase coverage by 0.4%.
The diff coverage is 86.4%.

@@           Coverage Diff            @@
##           master   #2053     +/-   ##
========================================
+ Coverage    73.3%   73.7%   +0.4%     
========================================
  Files         421     436     +15     
  Lines       15487   16061    +574     
  Branches     1563    1614     +51     
========================================
+ Hits        11357   11848    +491     
- Misses       3719    3774     +55     
- Partials      411     439     +28     

Flag	Coverage Δ
integrationtests	65.2% <84.1%> (+1.1%)	⬆️
unittests	67.0% <53.5%> (-0.6%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...s/models-library/src/models_library/basic_types.py	0.0% <0.0%> (ø)
...service-library/src/servicelib/rest_middlewares.py	64.2% <0.0%> (-1.4%)	⬇️
...ore_service_director_v2/utils/client_decorators.py	69.2% <ø> (ø)
...es/web/server/src/simcore_service_webserver/cli.py	73.8% <0.0%> (-5.7%)	⬇️
...er/src/simcore_service_webserver/security_roles.py	100.0% <ø> (ø)
...ages/models-library/src/models_library/projects.py	92.6% <66.6%> (-7.4%)	⬇️
...imcore_service_webserver/exporter/async_hashing.py	71.4% <71.4%> (ø)
.../src/simcore_service_webserver/storage_handlers.py	79.6% <78.0%> (-2.8%)	⬇️
...ore_service_webserver/exporter/request_handlers.py	80.0% <80.0%> (ø)
...rvice_webserver/exporter/formatters/base_models.py	82.9% <82.9%> (ø)
... and 36 more

[GitHK]

So after testing again, now it works! very cool!

• I found some issue after doing the following:

# this create a 10GB file in 1 second
fallocate -l 10G some_big_file.csv

• I added that in a project
• connected a jupyter-base-notebook where I downloaded the file
• without closing the project, I started an export which ended in a 400 with some timeout errors in the webserver:

2021-01-19T08:44:10.911222114Z DEBUG: simcore_service_webserver.diagnostics_core:is_sensing_enabled(69) - Diagnostics starts sensing after waiting 1566.95 secs [> 60.00 secs] since submodule init,
2021-01-19T08:44:10.899915631Z DEBUG: simcore_service_webserver.exporter.file_downloader:download_files(42) - Download results ['/tmp/ft9q0plx/38699ae2-5a30-11eb-b475-02420a000018/storage/0/38699ae2-5a30-11eb-b475-02420a000018/44fe431b-72de-43b2-80ad-9ec12b9d8f6e/notebooks.zip'],
2021-01-19T08:44:10.899933621Z (error(filepath_partial=<function Downloader.enqueue_file..filepath at 0x7f7d1fee5950>, url='http://172.17.0.1:9001/simcore/38699ae2-5a30-11eb-b475-02420a000018/4220704b-6c28-425d-8979-9959bfc105fa/some_big_file.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=12345678%2F20210119%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210119T083909Z&X-Amz-Expires=259200&X-Amz-SignedHeaders=host&X-Amz-Signature=378d7d70165c3d69cc5ee014814608e8c511b22dea1c7666e10c9af915303237', exception=TimeoutError())),
2021-01-19T08:44:10.899931541Z Errors:,
2021-01-19T08:44:10.912256385Z INFO: aiohttp.access:log(233) - 10.0.2.16 [19/Jan/2021:08:39:09 +0000] "GET /v0/projects/38699ae2-5a30-11eb-b475-02420a000018/export?compressed=true HTTP/1.1" 400 527 "-" "PostmanRuntime/7.26.8"

Should I have closed the project? well I guess that would have been better. at least for saving the notebook state...

So this is OK, thanks for pointing it out. Had to bump a timeout form 5 minutes. The reply is correct 400, as intended in case of error during the download. Too bad I cannot see the response from the request.

[sanderegg]

some minors. but great work. looking forward to using that feature.

[pcrespov]

this should be in the setup, as I mentioned above. We do not want to have globals. Instead we prefer singletons INSIDE the app. Nevermind, it is minor. I will change it later