Skip to content

🐛Autoscaling: Fix issue where all warm buffers would be used to replace hot buffers (🚨) #7079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Jan 23, 2025

What do these changes do?

After #6929 changes, a missing hot buffer would be replaced by a warm buffer if possible.
A bug in the implementation of this feature had for consequence that all the available warm buffers would be started to replace 1 missing hot buffer which leads to:

  • additional costs as too many machines are running
  • longer waiting times as all the warm buffers were used to replace 1 hot buffer and it takes time to re-create the missing warm buffers

driving test: test_warm_buffers_only_replace_hot_buffer_if_service_is_started_issue7071

@matusdrobuliak66 @YuryHrytsuk This should be hotfixed

Related issue/s

How to test

Dev-ops checklist

@sanderegg sanderegg added the a:autoscaling autoscaling service in simcore's stack label Jan 23, 2025
@sanderegg sanderegg added this to the Singularity milestone Jan 23, 2025
@sanderegg sanderegg self-assigned this Jan 23, 2025
@sanderegg sanderegg requested a review from pcrespov as a code owner January 23, 2025 17:58
@sanderegg sanderegg force-pushed the autoscaling/bugfix/scale-up-warm-buffer-too-much branch from f7447e3 to 711ff6d Compare January 23, 2025 17:59
Copy link

codecov bot commented Jan 23, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.19%. Comparing base (e0a69be) to head (87299c0).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7079      +/-   ##
==========================================
- Coverage   87.26%   87.19%   -0.08%     
==========================================
  Files        1634     1627       -7     
  Lines       64262    64074     -188     
  Branches     2051     2051              
==========================================
- Hits        56078    55868     -210     
- Misses       7848     7869      +21     
- Partials      336      337       +1     
Flag Coverage Δ
integrationtests 66.73% <ø> (-1.12%) ⬇️
unittests 85.83% <100.00%> (+0.27%) ⬆️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library 93.49% <ø> (ø)
pkg_dask_task_models_library 97.09% <ø> (ø)
pkg_models_library 91.43% <ø> (ø)
pkg_notifications_library 84.57% <ø> (ø)
pkg_postgres_database 88.41% <ø> (ø)
pkg_service_integration 70.18% <ø> (ø)
pkg_service_library 74.13% <ø> (ø)
pkg_settings_library 90.49% <ø> (ø)
pkg_simcore_sdk 85.38% <ø> (-0.12%) ⬇️
agent 96.45% <ø> (ø)
api_server 90.55% <ø> (ø)
autoscaling 96.10% <100.00%> (+<0.01%) ⬆️
catalog 90.32% <ø> (ø)
clusters_keeper 99.24% <ø> (ø)
dask_sidecar 91.26% <ø> (ø)
datcore_adapter 93.18% <ø> (ø)
director 76.92% <ø> (ø)
director_v2 91.27% <ø> (-0.02%) ⬇️
dynamic_scheduler 97.21% <ø> (ø)
dynamic_sidecar 89.75% <ø> (-0.04%) ⬇️
efs_guardian 90.39% <ø> (ø)
invitations 93.42% <ø> (ø)
osparc_gateway_server ∅ <ø> (∅)
payments 92.66% <ø> (ø)
resource_usage_tracker 89.33% <ø> (+0.38%) ⬆️
storage 89.57% <ø> (ø)
webclient ∅ <ø> (∅)
webserver 84.56% <ø> (-0.36%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0a69be...87299c0. Read the comment docs.

Copy link
Member

@mguidon mguidon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@sanderegg sanderegg changed the title 🐛Autoscaling: Fix issue where all warm buffers would be used to replace hot buffers 🐛Autoscaling: Fix issue where all warm buffers would be used to replace hot buffers (🚨) Jan 23, 2025
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx

@sanderegg sanderegg force-pushed the autoscaling/bugfix/scale-up-warm-buffer-too-much branch from 711ff6d to ff27450 Compare January 24, 2025 07:31
@sanderegg sanderegg merged commit 7f218ea into ITISFoundation:master Jan 24, 2025
85 of 91 checks passed
@sanderegg sanderegg deleted the autoscaling/bugfix/scale-up-warm-buffer-too-much branch January 24, 2025 10:58
matusdrobuliak66 pushed a commit that referenced this pull request Jan 24, 2025
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Mar 6, 2025
63 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:autoscaling autoscaling service in simcore's stack
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Autoscaling: all warm buffers are started to replace hot buffer
5 participants