everybody at z43 uses osparc for python/matlab scripting

[esraneufeld]

Niels wants everybody at z43 to use osparc in the context of their scripting. the idea is that this will give us a lot of feedback on usability.
currently, people write their own scripts all the time, but these scripts are rarely archived, shared, described. they can also not readily be applied to crunch large amounts of data due to the restriction of local machines.
the goal is to create an experience in osparc, where people will feel that they benefit from doing that work on osparc.

• in a first step, the team should come up with suggestions, how that experience could be achieved. i suggest starting with python and later shifting to matlab. points to consider will include:
• ease of shifting script to osparc
• ease of editing and rapidly reevaluating existing script in osparc (python notebooks?)
• ease of creating new script (notebook level, IDE level?)
• ease of organizing scripts (people might have many scripts, many variant of the same script, maybe a folder structure)
• ease of sharing scripts (publishing as a service); if there are many, some organization structure might also be needed
• ease of documenting what a script does, how it is used...
• ease of executing a script that requires access to locally stored data (mount drive?)
• ease of applying script to large amounts of data (crunching)

---

according to our discussion from today (May 12th)

• get the DALCO cluster
• s4l python version (already there?)
• sharing studies, services (not just templates) -> sharing not just a copy, but the actual thing (how are conflicts handled? non-concurrence initially)
• organization of scripts/projects (tree? catalog search?...) -> tags
• usage standard for z43 / guidelines
  -------------------------------------------------- (up to here: needed for release to z43?)
• versioning / branching
• matlab / octave
• launch isolve on osparc-cloud by pressing a button in s4l
• launch object as blackbox / viewable notebook / interactive notebook / editable code (related to fine-grained access rights issue)
• (bring process to data)

we need to be production level when we release it (in house)

[esraneufeld]

sylvain reboux would be valuable to include in defining and later testing the functionality.

[pcrespov]

State at Ueberwald milestone:

• In-depth review with SyR and EN on potential features based on z43 activities.
• First use-case defined: a python script runner [expected by mid next week]

[pcrespov]

Update Brown Islands sprint:

• Finishing alpha stage
  • Created and classified a list of possible features useful for Z43 [see slides]
  • python-script-runner user case resolved [demo]
  • Main features missing (in order of importance):
    • Machines at Z43 (next week)
    • S4L (ongoing)
    • Access to sandbox at Z43 (not started)
    • Export projects (ongoing: 1.5 sprint)
    • Matlab (will be based on python feature). Somebody must look at the license! (Nik checks)
    • New sidecars for dynamic services (not started: 2-3 sprints more)
• Beta stage shall start mid next sprint
  • Brownbag presentation to introduce osparc
  • first users (SP/PhD students?)

[drniiken]

usecases_z43usesOSPARC.pdf

[drniiken]

According to our meeting on 20200211, more details are provided within the use cases (see attached pdf).

[pcrespov]

Update on Dim Sum sprint for review

Current status

Implementing the following services:

• isolve as a computational service: drag&drop input file, connect solver, run, download; minimal licensing solution [WIP: integration missing]
• script runners as computational services [WIP: integration missing]
  • python
  • octave
  • matlab
• jupyterlab with different kernels as interactive services [WIP: integration missing]
  • python
  • rec5g (customized to Z43's python library)
  • octave
  • matlab [still missing]
• UI: one-click to jupyterlab (see Single view of Jupyter Notebooks in osparc osparc-simcore#1408 )

Ongoing development + Next steps

• intro selected users in Z43 (in-house extension of team black)
• integration of services above in osparc catalog :
  - due to upgrade of our integration workflow
  - now we deal with license/unlicense services
• matlab kernel for jupyterlab
• complete integration of supermash:
  • upgrade to py 3.7
  • industrialize upgrade and integration workflow!!!
• creation of service catalogs for deploys/groups (e.g. z43's catalog for staging)
• New sidecars for interactive services

[esraneufeld]

regarding the high priority items above:

• get the DALCO cluster
  should be clear. see also Move Dalco Cluster to Z43 #93
• Jupyter with s4l python version
  close to ready
• sharing studies, services (not just templates)
  still needed:
  • sharing of services (e.g., Jupyter notebook incl. the content of work-folder, without need for sharing the entire study)
  • ability of defining who has access to my study/service, without being a (super)admin (e.g., i want to share my script with thomas fussinger / to work with him on that script)
  • finer control on rights (view, execute, edit copy, edit master)
• organization of scripts/projects (tree? catalog search?...)
  • discussion was started with pedro for long term vision. define steps towards that (see Issue repository for studies/templates/services #252).
  • if many people share their scripts/notebooks, it has to be possible to find these easily, also, they cannot all be listed in the service catalog/templates catalogue. tags will not suffice.
    • a first solution could be a kind of directory/tree structure
    • another solution would probably combine search over names&meta-data, with a prioritization of results-display based on usage frequency (general and by the specific user)
• usage standard for z43 / guidelines
  should not be handled by developers. Katie and EN will organize it.
• launch isolve on osparc-cloud by pressing a button in s4l
  in this sprint, please discuss with s4l team how to realize this in the next spring.

note: is it a good decision to only save things in the work folder? should we not drop that one and save the entire root folder?

[pcrespov]

Update on C.Maurer sprint for review

Current status

• Focus on JupyterLab services with math kernels in python and octave
• BBL talk: Introduction of osparc to Z43 . The slides are available here. A demo covered the following points:
  • Dashboard
    • Cards: author, organization, description, ...
    • Tagging and filtering cards
  • Scripting Development Environment: jupyterlab customized for Z43
    • File explorer: with input, output, work dirs
    • Notebooks: half-code & half-document
    • Kernels: octave (matlab-like), python (+maths, +sim4life, … )
    • Integrated version control
    • Many more extras: interactive plots, slides, editor, markdown, ...
  • Studies: connecting your “program” to other “programs” via inputs/outputs
  • Share with everyone you like: export, collaborate, and publish (e.g. NIH-SPARC Portal website published simulations)
  • Integrated doc, feedback/issues

Ongoing development + Next steps

• Release to staging of first version of JupyterLab math (fix pending)
• Create customized JupyterLab with sim4life kernel in python
• Creation of a services catalog such that all services, as studies, can be also shared/classified/discovered etc

[pcrespov]

Update on Huo Guo sprint for review

Current status

• jupyterlab (updated): Released to staging of first version of JupyterLab math and new version in master.
• s4l python version (ongoing): Implemented customized JupyterLab with sim4life kernel in python (almost done)
• Designed mechanisms to version, discover and classify studies/services (ongoing):
  • Created classifiers repo including review/CI/CD (done). Ready to use in https://git.speag.com/oSparc/explore (done)
• sharing studies (ongoing). Implemented groups and lock mechanism.

Ongoing development + Next steps

• Finish customized JupyterLab with sim4life kernel in python
• Creation of a services catalog such that all services, as studies, can be also shared/classified/discovered etc