sim4life.io - WP4: Computational backend #950

mguidon · 2023-05-09T11:46:13Z

Description

With the latest version of sim4life.io, we are introducing an improved computational backend that ensures reliable and efficient job scheduling via the computational backend. Moving forward, all solver jobs will be scheduled via these facilities, enabling users to choose the hardware on which their jobs should run and providing the ability to inspect and operate on the job queue (subject to sufficient permissions).

This robust backend will be capable of handling 100s of concurrent jobs, ensuring that even the busiest periods will not cause any disruptions to service.

Furthermore, the backend functionality will also be made available through the API, allowing for integration with external systems (e.g. the sim4life desktop application) and further expanding the possibilities for users.

## Tasks
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4643
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4530
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4525
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/921
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/982
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/617
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/3999
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5094
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4524
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5073
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5074
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1196
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5000
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5293
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5436
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4880
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5336
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4526

### Enchanted Odyssey
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5493

### Schoggilebe
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5497
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5437
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5294
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5339
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5290
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1277
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5403

### This is Sparta!
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5218
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4727
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1218
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1219
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5251
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5203
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5237
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5261
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5264
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5149
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5252
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5287

### Kobayashi Maru
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5087
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1181
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5071
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5024
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5101
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5129
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/5146
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5108
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5120
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5141
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5147
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5155
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5162
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5164
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5163
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5165
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5167
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5195
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5204
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5201
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5076

### 7Peaks
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4159
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4958
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4781
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/621
- [x] Preferences: add preferences for max number of concurrent jobs
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1180
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4999
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/4975
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5008
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5010
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5013
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5042
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5054
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5018
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5025
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5031
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5026
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5032
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/5066

### Microhistory
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1034
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/4915
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/4930
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/3209

### Quilmes
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4517
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4756
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1126
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4376

### Sundae
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/4429
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4153
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4523

### Baklava
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4637

The text was updated successfully, but these errors were encountered:

sanderegg · 2023-05-14T18:57:37Z

Goal for sprint Pastel de Nata

progress on AppTeam Std Simulations, ideally run CF use-case
refactoring on computational backend, progress on separating PublicAPI calls from webserver load, return solver progress
progress on Public API missing entrypoints, and bug fixes
if possible progress on personalized resource limits

sanderegg · 2023-05-31T13:09:05Z

Update for sprint Pastel de Nata

Done:

🐛Computational backend: correct parsing of log lines osparc-simcore#4220
🎨 Rabbitmq/only listens to logs if needed osparc-simcore#4180
🐛service logs file not uploaded when service failing osparc-simcore#4242
♻️Sidecars: Image pulling is now a debug log, try to guess log level of services osparc-simcore#4232
✨ Store computational services progress in comp_tasks table, decouple webserver from api-server 🗃️ osparc-simcore#4197
🐛Api-server wrong start/stop/submit timestamps osparc-simcore#4254
🐛 passing data from a service to another fails osparc-simcore#4257
🐛Computational backend: disappearing logs osparc-simcore#4265
🐛Fix node progress value when the service did not run yet osparc-simcore#4277
✨RabbitMQ: add persistency on rabbitmq service osparc-simcore#4280
Ongoing:
✨🐛Webserver: enable socketio horizontal scaling + ensure only 1 update goes through (⚠️ devops) osparc-simcore#4286

Before on prod (05/31/2023) - all instances of webserver are loaded the same because they are coupled through logs/progress/db handling:

After on master (05/31/2023) - each instance is less coupled, still waiting for the ongoing PR to completely separate them:

mguidon · 2023-07-06T20:02:22Z

Update Watermelon

Done:

Personalized-resource-limits: User can now select desired resource requirements to run computational and dynamic services
Dask backend: Pass env from director-v2 to the worker/services osparc-simcore#4271
Dask backend: pass labels to be set on run containers osparc-simcore#4350

Ongoing:

Robustness improvements/refactoring

sanderegg · 2023-08-09T12:15:28Z

Update Sundae

Done:

bugfixes #4153
connection of computational backend to resource usage tracking service #4523
new clusters keeper service to automatically create computational clusters in AWS #4591

Ongoing:

connect clusters-keeper service to oSparc and create computational clusters on the fly
API framework for iSolve #621
AppTeam Std Simulations on S4L/AWS #885
Robustness improvements/Bugfixes from usage

sanderegg · 2023-09-06T14:55:13Z

Update Baklava

Done:

Computational Backend: CompRuns validation error osparc-simcore#4637

Ongoing:

Computational Backend: management and autoscaling of external clusters osparc-simcore#4159
- Dask backend: Create a separate cluster for group automatically osparc-simcore#4521
- Dask backend: autoscaling in osparc-dask-scheduler osparc-simcore#4522

sanderegg · 2023-09-06T15:12:04Z

The below schema shows the overall architecture for the on-demand clusters.
Some important points here are:

the computational clusters are created per user/wallet
in case of maintenance in simcore, these clusters shall be able to continue running independently

sanderegg · 2023-09-22T17:37:46Z

Update Quilmes

Done:

Dask backend: Create a separate cluster for group automatically osparc-simcore#4521
Computational backend: state is reported as Pending even though the computation already started and logs are shown osparc-simcore#4517
UI says that a comp. service is running, while it is not true osparc-simcore#4756
Show timestamps in the output log file for computational services #1126
Computational backend: service shall be able to define how progress is defined in the service osparc-simcore#4376

Ongoing:

Dask backend: autoscaling in osparc-dask-scheduler osparc-simcore#4522

sanderegg · 2023-10-31T10:33:48Z

Update Microhistory

Done and working

Separate cluster is created for each set of user/wallet combination on demand in Amazon AWS,
Cluster is a primary machine (t2.micro), on which a stack containing dask-scheduler, autoscaling, redis, dask-sidecar services is started, dask-sidecar only runs on worker machines,
autoscaling service creates 1 worker machine (g4dn.xlarge),
Only computational services that use a pricing unit defined as g4dn.xlarge machine can run,
computational service uses the all the resources provided by the machine (a bit less than 16Gb/4CPUs)

--> Running computational service should work for one service at a time, provided they are set up to use a g4dn.xlarge machine type, there is no upscaling of the machines so parallel jobs will have to wait in line (if multiple isolve jobs are sent, they will be executed one after the other).

should work in 3 weeks

Cluster shall create correct machine based on plan (not just g4dn.xlarge), so potentially better machine fit/performance,
identify computational child jobs (for example started from s4l) and show them in UI
maybe upscaling of separate cluster (needs discussions on how to do it, it has influence on costs, etc)

should not be available in 3 weeks

upscaling?
optimisations

sanderegg · 2023-11-28T16:32:11Z

Update 7peaks

Summary

It is now possible to run computational services on their required AWS instance types. Also child computational job logs show up in the logs of the parent service (e.g. sim4life/jupyterlab starting a computational job).
Upscaling is still not implemented.

Done

Ongoing

bugfixing
improvements on user feedback (cluster status, number of machines, etc...)

sanderegg · 2024-01-07T19:08:40Z

Update Kobayashi Maru

Summary

bugfixes:
- handling of on-demand computational clusters (timeouts, reported states)
- concurrent computing of tasks
monitoring & manual interventions:
- CLI tool to monitor on-demand computational clusters and dynamic service machines
- partially clear jobs in a specific cluster
- allow tracing of created machines via tags on EC2 instances

Done ✅

various fixes for GPU-based computational services on multi-GPU machines
migration of sleepers test to Playwright framework to have more reliable and more flexible E2E testing and compatibility with on-demand computational clusters
various fixes regarding invalid state reported by the computational clusters
added timeout in case of non responding cluster for more than 10 minutes
improvement of response time when retrieving the computational clusters state via Public API
new CLI-based monitoring tool to check current state of auto-scaled EC2 instances and their running states

Problematic issues (being worked on) 🚧

Tasks not optimaly distributed to all machines
Machines not properly shut down - unnecesary costs:
- Computational clusters: autoscaling unable to scale down worker #1219
- Computational clusters: "erred" jobs prevent workers from being shutdown #1218

Open Features 🚧

sanderegg · 2024-01-30T20:12:59Z

Update This is Sparta!

Summary

Multi-processing is now working as expected (10 machines will take 10 jobs)
Added facilities for tracing jobs/logs
Secure transmission of data in the computational backend
Stability improvements

Done ✅

Tasks not optimaly distributed to all machines
Machines not properly shut down - unnecesary costs:
- Computational clusters: autoscaling unable to scale down worker #1219
- Computational clusters: "erred" jobs prevent workers from being shutdown #1218
Secure connection to external computational clusters osparc-simcore#4727

Ongoing 🚧

Open Features 🚧

sanderegg · 2024-03-21T15:42:05Z

Update Schoggilebe

Summary

Using Ansible to create AMIs for AWS (reproducibility of machine images) for both dynamic/computational autoscaled machiens
Authorize deployment of computational cluster in different AWS regions
Improve labeling of machines, network and volumes for better cost management

Done

Autoscaling/Clusters-keeper: label volumes and network interfaces osparc-simcore#5403
Computational clusters: when the EC2 vCPU quota is reached the clusters does not start processing #1277
Clusters-keeper: WORKERS_EC2_INSTANCES_ALLOWED_TYPE is not passing 16KB limit for EC2 instances osparc-simcore#5290
Autoscaling: use labels instead of draining nodes osparc-simcore#5339
create new updated AMIs for autoscaling based on Ansible osparc-simcore#5294
External clusters: jobs sent at interval do not get a machine even if there is still room osparc-simcore#5437

Ongoing

mguidon added PO issue Created by Product owners s4l:web sim4life product in osparc.io labels May 9, 2023

mguidon mentioned this issue May 9, 2023

sim4life.io #878

Closed

mguidon changed the title ~~sim4life.io - Computational backend~~ sim4life.io - WP4: Computational backend May 9, 2023

pcrespov assigned pcrespov, mguidon and sanderegg May 12, 2023

pcrespov added this to the Pastel de Nata milestone May 12, 2023

sanderegg mentioned this issue Jun 8, 2023

✨Personalized resource limits: add DB table for max resources per user 🗃️ ITISFoundation/osparc-simcore#4335

Merged

2 tasks

pcrespov modified the milestones: Pastel de Nata, Watermelon Jun 12, 2023

GitHK modified the milestones: Watermelon, Sundae Jul 24, 2023

matusdrobuliak66 assigned GitHK and matusdrobuliak66 and unassigned pcrespov Jul 24, 2023

sanderegg assigned pcrespov, mrnicegyu11 and YuryHrytsuk and unassigned mrnicegyu11 and YuryHrytsuk Jul 24, 2023

matusdrobuliak66 modified the milestones: Sundae, Baklava Aug 22, 2023

sanderegg assigned bisgaard-itis Aug 22, 2023

sanderegg assigned elisabettai and unassigned GitHK, pcrespov and bisgaard-itis Sep 18, 2023

sanderegg modified the milestones: Baklava, the nameless Sep 18, 2023

mrnicegyu11 modified the milestones: the nameless, Kobayashi Maru Nov 27, 2023

sanderegg modified the milestones: Kobayashi Maru, This is Sparta! Jan 11, 2024

ignapas assigned GitHK Jan 30, 2024

bisgaard-itis removed this from the This is Sparta! milestone Feb 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sim4life.io - WP4: Computational backend #950

sim4life.io - WP4: Computational backend #950

mguidon commented May 9, 2023 •

edited by sanderegg

Loading

sanderegg commented May 14, 2023 •

edited

Loading

sanderegg commented May 31, 2023 •

edited

Loading

mguidon commented Jul 6, 2023 •

edited by sanderegg

Loading

sanderegg commented Aug 9, 2023 •

edited

Loading

sanderegg commented Sep 6, 2023 •

edited

Loading

sanderegg commented Sep 6, 2023 •

edited

Loading

sanderegg commented Sep 22, 2023 •

edited by bisgaard-itis

Loading

sanderegg commented Oct 31, 2023 •

edited

Loading

sanderegg commented Nov 28, 2023 •

edited

Loading

sanderegg commented Jan 7, 2024 •

edited

Loading

sanderegg commented Jan 30, 2024 •

edited

Loading

sanderegg commented Mar 21, 2024

sim4life.io - WP4: Computational backend #950

sim4life.io - WP4: Computational backend #950

Comments

mguidon commented May 9, 2023 • edited by sanderegg Loading

Description

sanderegg commented May 14, 2023 • edited Loading

Goal for sprint Pastel de Nata

sanderegg commented May 31, 2023 • edited Loading

Update for sprint Pastel de Nata

mguidon commented Jul 6, 2023 • edited by sanderegg Loading

Update Watermelon

sanderegg commented Aug 9, 2023 • edited Loading

Update Sundae

sanderegg commented Sep 6, 2023 • edited Loading

Update Baklava

sanderegg commented Sep 6, 2023 • edited Loading

sanderegg commented Sep 22, 2023 • edited by bisgaard-itis Loading

Update Quilmes

sanderegg commented Oct 31, 2023 • edited Loading

Update Microhistory

Done and working

should work in 3 weeks

should not be available in 3 weeks

sanderegg commented Nov 28, 2023 • edited Loading

Update 7peaks

Summary

Done

Ongoing

sanderegg commented Jan 7, 2024 • edited Loading

Update Kobayashi Maru

Summary

Done ✅

Problematic issues (being worked on) 🚧

Open Features 🚧

sanderegg commented Jan 30, 2024 • edited Loading

Update This is Sparta!

Summary

Done ✅

Ongoing 🚧

Open Features 🚧

sanderegg commented Mar 21, 2024

Update Schoggilebe

Summary

Done

Ongoing

mguidon commented May 9, 2023 •

edited by sanderegg

Loading

sanderegg commented May 14, 2023 •

edited

Loading

sanderegg commented May 31, 2023 •

edited

Loading

mguidon commented Jul 6, 2023 •

edited by sanderegg

Loading

sanderegg commented Aug 9, 2023 •

edited

Loading

sanderegg commented Sep 6, 2023 •

edited

Loading

sanderegg commented Sep 6, 2023 •

edited

Loading

sanderegg commented Sep 22, 2023 •

edited by bisgaard-itis

Loading

sanderegg commented Oct 31, 2023 •

edited

Loading

sanderegg commented Nov 28, 2023 •

edited

Loading

sanderegg commented Jan 7, 2024 •

edited

Loading

sanderegg commented Jan 30, 2024 •

edited

Loading