werft rarely runs a job twice in one run.

[utam0k]

Bug description

https://github.com/gitpod-io/gitpod/pull/12793/files#r966677722

You can see in this log that the wait to build the preview environment should be one time, but it appears twice for some reason
e.g. https://werft.gitpod-dev.com/job/gitpod-custom-to-test-pfw.137/logs

Perhaps it is a network condition issue. It might be related to this.
#12629

Steps to reproduce

I don't know

Workspace affected

No response

Expected behavior

always try to make only one run

Example repository

No response

Anything else?

No response

[mads-hartmann]

I have seen something similar for other long running jobs (don't have a link at hand I'm afraid). My theory is the following:

Werft is just tail'ing logs from the pods that are running (AFAIK) so I suspect that it gets confused for some reason (e.g. maybe it looses the connection to the pod) and tails the logs from beginning again - this makes it appear as if the job started over from the beginning. So I think this is a UI problem rather than Werft running the job twice ☺️

This would be easier to verify if we had timestamps on the logs.

[iQQBot]

Not really, it actually just runs twice, but it's not a complete run either

[iQQBot]

We only commit and push once at the beginning of the script, but if you look at some test-branch, you'll see that the same commit is committed twice, and that the commits are stacked, indicating that the script is executed twice within the pod

[iQQBot]

that's why I adding these code

https://github.com/gitpod-io/gitpod/blob/main/.werft/ide-integration-tests-startup.yaml#L69-L71

[mads-hartmann]

We only commit and push once at the beginning of the script, but if you look at some test-branch, you'll see that the same commit is committed twice

@iQQBot Oh wow, that's unexpected! Would you be able to do a quick "echo" before exit 0 so that we can grab the logs and get a sense of how often this happens and possibly dive a bit deeper when it does

[iQQBot]

This bug doesn't happen consistently, but I'll give it a try

[mads-hartmann]

I think I found the problem when debugging something else. See this comment https://github.com/gitpod-io/ops/issues/5720#issuecomment-1269949085