try harder when attempting to add or remove annotation from workspaces

[sagor999]

Description

In GCP logs I observed this error pop up quite often:
cannot workspace: cannot mark workspace redacted with -gitpod/never-ready: Operation cannot be fulfilled on pods "ws-redacted": the object has been modified; please apply your changes to the latest version and try again
https://cloudlogging.app.goo.gl/d9NjYg6x9NgPfumeA

This is caused by default backoff being too short and after 4 attempts to update pod with updated annotation, it fails and marks whole workspace as failed.
So instead this PR make it try harder.

Related Issue(s)

Fixes #

How to test

There is no easy way to test this, as it happens when some other process updates pod while we are trying to call markWorkspace function.

Release Notes

Try harder to update workspace annotation to prevent workspace from being marked as failed erroneously.

Documentation

[codecov]

Codecov Report

Merging #8519 (31ef3bd) into main (6db0f39) will increase coverage by 22.14%.
The diff coverage is 45.45%.

@@             Coverage Diff             @@
##             main    #8519       +/-   ##
===========================================
+ Coverage   11.17%   33.32%   +22.14%     
===========================================
  Files          18       31       +13     
  Lines         993     4585     +3592     
===========================================
+ Hits          111     1528     +1417     
- Misses        880     2941     +2061     
- Partials        2      116      +114     

Flag	Coverage Δ
components-gitpod-cli-app	11.17% <ø> (ø)
components-ws-manager-app	39.44% <45.45%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
components/ws-manager/pkg/manager/create.go	82.95% <ø> (ø)
components/ws-manager/pkg/manager/manager.go	20.28% <0.00%> (ø)
components/ws-manager/pkg/manager/monitor.go	9.46% <0.00%> (ø)
components/ws-manager/pkg/manager/annotations.go	68.42% <100.00%> (ø)
...-manager/pkg/manager/internal/workpool/workpool.go	100.00% <0.00%> (ø)
...s/ws-manager/pkg/manager/internal/grpcpool/pool.go	71.83% <0.00%> (ø)
components/ws-manager/pkg/clock/clock.go	68.62% <0.00%> (ø)
components/ws-manager/pkg/manager/status.go	74.77% <0.00%> (ø)
components/ws-manager/pkg/manager/probe.go	0.00% <0.00%> (ø)
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 362f860...31ef3bd. Read the comment docs.