Skip to content

KafkaMessageSource Blocks on Initial Poll When No Records #3637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
garyrussell opened this issue Sep 30, 2021 · 0 comments · Fixed by #3638
Closed

KafkaMessageSource Blocks on Initial Poll When No Records #3637

garyrussell opened this issue Sep 30, 2021 · 0 comments · Fixed by #3638
Assignees
@garyrussell
Labels
in: Kafka type: bug
Milestone
5.5.5

Comments

@garyrussell
Copy link
Contributor

5.5.4.RELEASE

KMS increases the poll timeout for the first poll (assignment).

this.assignTimeout =
		Duration.ofMillis(Math.max(this.pollTimeout.toMillis() * 20, MIN_ASSIGN_TIMEOUT)); // NOSONAR - magic

With the default pollTimeout (5s), when there are no records, the poll blocks for 100s.

This causes a delay, for example, with a PollableConsumer in SCSt the poll is not responsive.

Wake the consumer when the partitions are assigned.
@garyrussell garyrussell added this to the 5.5.x milestone Sep 30, 2021
@garyrussell garyrussell self-assigned this Sep 30, 2021
@garyrussell garyrussell modified the milestones: 5.5.x, 5.5.5 Sep 30, 2021
garyrussell added a commit to garyrussell/spring-integration that referenced this issue Sep 30, 2021
@garyrussell
spring-projectsGH-3637: Fix KafkaMessageSource First Poll
4eb5c11 
Resolves spring-projects#3637

If there are no records to receive, the poll blocked for `20*pollTimeout`.

Wake the consumer during partition assignment; however, this changes the behavior
when there are records present - the first poll always returns no records.

Detect that the wakeup was due to assignment and perform another poll.

**cherry-pick to 5.4.x**
garyrussell added a commit to garyrussell/spring-integration that referenced this issue Sep 30, 2021
@garyrussell
spring-projectsGH-3637: Fix KafkaMessageSource First Poll
5913520 
Resolves spring-projects#3637

If there are no records to receive, the poll blocked for `20*pollTimeout`.

Wake the consumer during partition assignment; however, this changes the behavior
when there are records present - the first poll always returns no records.

Detect that the wakeup was due to assignment and perform another poll.

Also add some debug logging, and don't create a new consumer if the source was
stopped.

**cherry-pick to 5.4.x**
@garyrussell garyrussell mentioned this issue Sep 30, 2021
Merged
artembilan pushed a commit that referenced this issue Sep 30, 2021
@garyrussell @artembilan
GH-3637: Fix KafkaMessageSource First Poll
b8e414d 
Resolves #3637

If there are no records to receive, the poll blocked for `20*pollTimeout`.

Wake the consumer during partition assignment; however, this changes the behavior
when there are records present - the first poll always returns no records.

Detect that the wakeup was due to assignment and perform another poll.

Also add some debug logging, and don't create a new consumer if the source was
stopped.

**cherry-pick to 5.4.x**
artembilan pushed a commit that referenced this issue Sep 30, 2021
@garyrussell @artembilan
GH-3637: Fix KafkaMessageSource First Poll
8a5c037 
Resolves #3637

If there are no records to receive, the poll blocked for `20*pollTimeout`.

Wake the consumer during partition assignment; however, this changes the behavior
when there are records present - the first poll always returns no records.

Detect that the wakeup was due to assignment and perform another poll.

Also add some debug logging, and don't create a new consumer if the source was
stopped.

**cherry-pick to 5.4.x**
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@garyrussell garyrussell

Labels
in: Kafka type: bug
Projects
None yet
Milestone
5.5.5
Development

Successfully merging a pull request may close this issue.

GH-3637: Fix KafkaMessageSource First Poll garyrussell/spring-integration
2 participants
@garyrussell @artembilan