Skip to content

[PVC] Workspace fails on restart with misconfigured initializer if some prior instance "failed" #14030

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
geropl opened this issue Oct 20, 2022 · 4 comments · Fixed by #14053
Closed
Assignees
Labels
component: server team: workspace Issue belongs to the Workspace team type: bug Something isn't working

Comments

@geropl
Copy link
Member

geropl commented Oct 20, 2022

context: https://gitpod.slack.com/archives/C047BR51GDR

especially thread: https://gitpod.slack.com/archives/C047BR51GDR/p1666257320068649

example workspaces:

  1. gitpodio-gitpod-njdud87zz0n https://gitpod.io/admin/workspaces/gitpodio-gitpod-njdud87zz0n
  2. gitpodio-gitpod-0crlhy9mhdk https://gitpod.io/admin/workspaces/gitpodio-gitpod-0crlhy9mhdk

example trace for 1. containing wrong initializer: https://jaeger-eu-prod.workspace-preview.gitpod-io-dev.com/trace/71c1f5f3e97f870a

Code:

  1. here we set hasValidBackup to false if any prior instance failed
  2. here volumeSnapshots is undefined
  3. so we end up here, where we create a BackupInitializer, but without setting FromVolumeSnapshot to true and without setting VolumeSnapshot.

/cc @sagor999

@geropl geropl added type: bug Something isn't working component: server team: workspace Issue belongs to the Workspace team labels Oct 20, 2022
@jenting jenting changed the title PVC-Workspace fails on restart with misconfigured initializer if some prior instance "failed" [PVC] Workspace fails on restart with misconfigured initializer if some prior instance "failed" Oct 20, 2022
@iQQBot
Copy link
Contributor

iQQBot commented Oct 20, 2022

here we set hasValidBackup to false if any prior instance failed

Isn't this set to true if any prior instance success? then it got the wrong instance id, and set it to volumeSnapshotId

but this instance is failed, so there is no volumeSnapshots were found

@geropl
Copy link
Member Author

geropl commented Oct 20, 2022

@iQQBot Looking at this again, you're absolutely right. Well, should be easy to fix then. Will create a PR. 👍

@geropl geropl moved this to Scheduled in 🍎 WebApp Team Oct 20, 2022
@geropl geropl moved this from Scheduled to In Progress in 🍎 WebApp Team Oct 20, 2022
@geropl geropl self-assigned this Oct 20, 2022
@sagor999
Copy link
Contributor

Agreed, this seems like a bug.
Intention originally was to make sure that we do not attempt to restore from failed build (as there would be no backup\snapshot available).

@kylos101
Copy link
Contributor

@geropl Removed workspace team project from this (as its shipping in server). Feel free to add back if anything else is needed?

Repository owner moved this from In Progress to In Validation in 🍎 WebApp Team Oct 24, 2022
@geropl geropl moved this from In Validation to Done in 🍎 WebApp Team Oct 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: server team: workspace Issue belongs to the Workspace team type: bug Something isn't working
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

4 participants