Try to send entire batch before returning any MessageTooLargeErrors in Writer.WriteMessages

[Evanjt1]

Currently, Writer.WriteMessages returns early if it encounters a message that is too large:

kafka-go/writer.go

Line 322 in 5e37e04

err := MessageTooLargeError{

Since messages may have already been added to w.msgs before encountering the MessageTooLargeError, wouldn't it be better to try and submit all messages in the batch before returning this error? I have a feeling that the early return is due to only one error being returned from a WriteMessages call so ensuring this error gets returned allows the caller to handle appropriately.

Is there some other edge case that I'm missing here for why the receiver returns early when encountering this issue?

Linking #363 since it's related to the singular return from WriteMessages

Obligating myself to implement this, if needed.

[achille-roussel]

Hey @Evanjt1, thanks for raising the question.

You're right that the current behavior isn't ideal, I think we should either attempt to send as many messages as possible (all messages that aren't too large), or do a check prior to sending the messages and either send them all or send nothing.

@stevevls you may have an opinion on this as well.

[Evanjt1]

@achille-roussel I typically lean towards processing as much as possible and rejecting only the malformed/too large messages rather than failing out the entire batch. I perused a few clients for other systems that have similar batch send, and that seems to be the preferred way to go.

[achille-roussel]

This issue has been stale for a while. I'll go ahead and close it but feel free to reopen if this is still a problem.

[stevevls]

Oops...I didn't realize I was flagged on this issue because I was on paternity leave last October. 😅 Anyhow, I think that some clients may be sensitive to message order within a partition, so the safest option is the current behavior. If we wanted to enable sending all the messages in a single batch, then I would opt for a configuration option to govern the behavior. Just putting my thoughts down here in case we reopen the issue down the line. 😄