Skip to content

On demand aws master #944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
22 tasks
YuryHrytsuk opened this issue Jan 27, 2025 · 1 comment
Open
22 tasks

On demand aws master #944

YuryHrytsuk opened this issue Jan 27, 2025 · 1 comment
Assignees
@YuryHrytsuk
Milestone
Bazinga!

Comments

@YuryHrytsuk
Copy link
Collaborator

YuryHrytsuk commented Jan 27, 2025 •
edited
Loading

Proposal

Stopping

  • enable redis maintenance logout
    • fail if redis key is already set (add boolean flag to override this behavior)
    • end date of maintenance? Which value to choose?
  • wait until sidecars are shutdown with timeout
    • if some fail stop the shutdown procedure
    • habe a boolean flag that tells "continue even if dy-sidecars cannot terminate gracefully ~~ lose data"
    • report on non-terminated dy-sidecars (have means to keep this information)
  • wait until autoscaled EC2 are terminated
    • if some fail, stop the procedure until special flags allows hanging Autoscaled EC2
    • report on non-terminated autoscaled ec2 (have means to keep this information)
  • Remove all swarm stacks
  • Stop static machines

Starting

  • start machines
  • deploy Swarm Stacks
  • remove maintenance key

Extra:

  • disable all e2e tests when deployment is off

Nuances:

  • What happens to AWS NLB
  • Data on hanging sidecars is going to be lost (--> no shutdown if sidecar is hanging?)
    • Will it? If EC2 is stopped no data loss shall occur

Extra requirements

All procedures

  • Can be run manually

Stopping Usage Usecases

  • can be run by backenders on demand
  • idempotent

Starting Usage Usecases

  • can be done by backenders on demand
  • automatically scheduled after automatic stopping
  • idempotent

Possible use cases

  • shutdown procedure has started but deployment is already under maintenance (via redis and / or maintenance pages stack)
  • shutdown procedure is completed but developers suddenly need aws master
  • there is a special week when aws master needs to be up all the time
  • shutdown procedure has started while deployment is down (e.g. Hardware issues)
  • shutdown procedure fail to complete
@YuryHrytsuk YuryHrytsuk transferred this issue from ITISFoundation/osparc-issues Jan 27, 2025
@YuryHrytsuk YuryHrytsuk self-assigned this Jan 27, 2025
@YuryHrytsuk YuryHrytsuk added this to the Singularity milestone Jan 27, 2025
@sanderegg sanderegg modified the milestones: Singularity, The Awakening Feb 24, 2025
@YuryHrytsuk YuryHrytsuk mentioned this issue Mar 31, 2025
Merged
1 task
YuryHrytsuk added a commit to YuryHrytsuk/osparc-ops-environments that referenced this issue Apr 29, 2025
@YuryHrytsuk
Expose dynamic scheduler API
704d07b 
API is exposed under `/v1` path. Here we build traefik Router to point
traffic to this endpoint

Related PR(s):
* ITISFoundation/osparc-simcore#7454

Related Issue(s):
* ITISFoundation#944
@YuryHrytsuk YuryHrytsuk mentioned this issue Apr 29, 2025
Merged
1 task
YuryHrytsuk added a commit that referenced this issue Apr 29, 2025
@YuryHrytsuk
Expose dynamic scheduler API (#1035)
b2b4d6a 
API is exposed under `/v1` path. Here we build traefik Router to point
traffic to this endpoint

Related PR(s):
* ITISFoundation/osparc-simcore#7454

Related Issue(s):
* #944
This was referenced May 5, 2025
Merged
Merged
@YuryHrytsuk YuryHrytsuk modified the milestones: Pauwel Kwak, Bazinga! May 6, 2025
@YuryHrytsuk YuryHrytsuk mentioned this issue May 14, 2025
Closed
2 tasks
@YuryHrytsuk
Copy link
Collaborator Author

Implemented in https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1381

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@YuryHrytsuk YuryHrytsuk

Labels
None yet
Projects
None yet
Milestone
Bazinga!
Development

No branches or pull requests
2 participants
@sanderegg @YuryHrytsuk