Patch of resource fails due to 'the server rejected our request due to an error in our request'

[tomdw]

Bug Report

What did you do?

Written a reconciler for a Kubernetes resource which patches the resource during cleanup (probably to remove the finalizer).

What did you expect to see?

Succesfull patch of the resource without any ERROR log

What did you see instead? Under which circumstances?

The Patch logs an ERROR due to a 'the server rejected our request due to an error in our request' error from kubernetes, but t the same time the cleanup continues and seems to remove the resource. If it succeeds I do not expect an ERROR log.

ERROR Error during event processing ExecutionScope{ resource id: ResourceID{name='my-resource', namespace='my-namespace'}, version: 2616628} [ X my-resource in my-namespace ]
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://192.168.194.129:443/apis/be.mygroup/v1/namespaces/my-namespace/x/my-resource. Message: the server rejected our request due to an error in our request. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[], group=null, kind=null, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=the server rejected our request due to an error in our request, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}). 
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:205) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397) 
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:764) 
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:231) 
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:236) 
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.edit(HasMetadataOperation.java:65) 
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.edit(HasMetadataOperation.java:44) 
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher$CustomResourceFacade.patchResourceWithoutSSA(ReconciliationDispatcher.java:420) 
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.conflictRetryingPatch(ReconciliationDispatcher.java:368) 
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleCleanup(ReconciliationDispatcher.java:266) 
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:95) 
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:68) 
	at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:467) 
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
	at java.base/java.lang.Thread.run(Unknown Source) 
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://192.168.194.129:443/apis/be.mygroup/v1/namespaces/my-namespace/x/my-resource. Message: the server rejected our request due to an error in our request. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[], group=null, kind=null, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=the server rejected our request due to an error in our request, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549) 
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source) 
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:141) 
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) 
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) 
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) 
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) 
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) 
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at io.fabric8.kubernetes.client.vertx.VertxHttpRequest.lambda$consumeBytes$1(VertxHttpRequest.java:120) 
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270) 
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:252) 
	at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:76) 
	at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250)
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:421) 
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:279) 
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:157) 
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:731)
	at io.vertx.core.impl.ContextImpl.lambda$execute$7(ContextImpl.java:329) 
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) 
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) 
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) 
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) 
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) 
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 
	... 1 common frames omitted | plugin-transformer-cloudfunction-operator

Environment

Kubernetes cluster type:

vanilla

$ Mention java-operator-sdk version from pom.xml file

5.0.4

$ java -version

21

$ kubectl version

1.30.7

Possible Solution

Might be related to which kind of patch is used? See https://stackoverflow.com/questions/57480205/error-while-applying-json-patch-to-kubernetes-custom-resource where a similar problem is described and a merge patch is to be used to avoid such a thing?

Additional context

N/A

[csviri]

Hi @tomdw , could you pls create a simple reproducer, we did not see this before, and the finalizer removal is covered in quite a lot of tests. So would be good to see the details.

[afalhambra-hivemq]

I randomly got the same error and stacktrace when patching the CR status in a regular reconciliation loop (no resource clean-up call), but only when SSA is disabled for patching the primary resource.

.withUseSSAToPatchPrimaryResource(false)
.withSSABasedCreateUpdateMatchForDependentResources(true)

Will try to provide you with a reproducer

[csviri]

Thx, just a note we don't do anything special there, just calling the client that calls the API.

Note that a common issues is when you mix ssa patch with non ssa patch on resources, make sure you settle on one or the other.

[afalhambra-hivemq]

Note that a common issues is when you mix ssa patch with non ssa patch on resources, make sure you settle on one or the other.

mmm, right, then why not just keep one single entry point/method to configure SSA? Otherwise having two withSSAxxx method may lead to wrong configuration, or?

[csviri]

We wanted to be backwards compatible in this case. Also not that patching the primary resource is part of the low level API, and DependentResources is something that you might use or not. Also there is difference between managing patching primary and some other resource, those does not effect each other.

So this level of granularity seems to right for me.

[csviri]

Note that a common issues is when you mix ssa patch with non ssa patch on resources, make sure you settle on one or the other.

mmm, right, then why not just keep one single entry point/method to configure SSA? Otherwise having two withSSAxxx method may lead to wrong configuration, or?

Ahh sorry, misunderstanding, this is how I mean it: if you mix that on same resource. Thus had a resource managed before without SSA and you switch manage to SSA of that same resource.

[afalhambra-hivemq]

Understood. Thanks for clarifying @csviri.
Just to be clear here in this case.
I get random failures with the same 422 error code and same the server rejected our request due to an error in our request error message with the following operator configuration:

.withUseSSAToPatchPrimaryResource(false)
.withSSABasedCreateUpdateMatchForDependentResources(true)

Stacktrace in my case is slight different since for me this is not happening when cleaning-up a resource but in a regular reconciliation loop.