fix: catch exceptions from updating the status

[shawkins]

There are two possible handlings here:

• catch and log the error from the status patch, so that the original retry handling is preserved
• log the original exception, and let the error from the status patch take over as it does now

I opted for the former, but I'm open to the latter.

closes: #2751

[csviri]

log the original exception, and let the error from the status patch take over as it does now

This makes more sense to me, considering the following situation having a patch from error handler that fails and not to retry raises some concerns of consistency. This way might be at least eventual consistent in general. But not retrying lead to an unexpected state: reconciliation failed and even status is not updated.
What do you think?

[shawkins]

This makes more sense to me, considering the following situation having a patch from error handler that fails and not to retry raises some concerns of consistency.

Why would you not retry? Wouldn't that mean the core reconciliation action has no retry or retires left?

An independent retry for patching the status doesn't make as much sense for me. Since there is already retry logic built in the client, the most likely causes for the patching to fail would be:

• a 409 - in which case I'd expect another event to come in
• the api server is down for longer than the client retry, which I don't think requires special handling just for patching the status

[csviri]

Ahh yes, so if there is explicit no retry we could say, we don't retry even if patch fails. But again, I guess some might want to patch status for a reason, but you probably right so we might be good enough with logging in this case.

other opinions? @metacosm @xstefank

[xstefank]

It makes sense to me like this. If there are requests to patch the status, we could put this under config.

[metacosm]

Ideally, it would make sense to always try to patch the status but it might not be possible, especially if it already failed at least once…

[csviri]

Thank you @shawkins !