-
Notifications
You must be signed in to change notification settings - Fork 812
Handling 0
as possible required ack value and InvalidRequiredAcks
errors
#176
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we have to check that this is a produce request?
do
is used for all operations, but "required acks" is only employed when send messages isn't it?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I totally agree with you but the
waitResponse
is done at connection level (so in the conn.go file) where as far as I can see from the code we don't have information if it's a producer or a consumer request (which is at higher level). We should do a refactoring for moving the waiting for the response at the upper layer.Is that really a problem? I mean, the conn is always initialized with -1 as a default value so other requests will work out of the box (and for consumer we cannot change it).
Btw if you think there is a simple way to check for producer request, feedback are really appreciated :-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just adding that if the conn is there for representing a connection to the broker, the
requiredAcks
should not be an information at this so low level imho ... but it was already there.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could thing to add a new
waitResponse bool
parameter for thedo
function passed by thewriteOperation
andreadOperation
used by higher level protocol operation for which we know if it's needed to wait a response or not.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we're pretty much forced to handle it at this level since it's core to how the kafka protocol works (some requests have responses, some don't, currently we only support the former), and we use the
Conn
abstraction for all interactions with kafka brokers.We don't know how the connection is being handled, but we do know what kafka API call is being made, so I think we can figure out whether the "required ack" applies or not based on that, can't we?
It's OK to modify internal APIs to support new use cases as well, if we need to pass more info down to this method then let's do it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Which information inside conn.go provides the information about which API is called. Isn't it an information of higher level?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The entry point for producing messages is here https://github.com/segmentio/kafka-go/blob/master/conn.go#L828
So this is where we have most of the context for deciding whether waiting for a ack is required or not.
I think the simplest is going to be copying the content of the
do
method intoWriteCompressedMessages
, in replacement for the call towriteOperation
, and based on whether "required acks" are set or not we wait for the response.What do you think about this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ppatierno do you still have interest in pushing this through the finish line?