kafka_group unexpected error="undefined method `try'

[den-is]

Can't connect and messages using kafka_group.
To enhance performance I wanted it to have 10 partitions for the topic and connect 5-10 fluentd consumers.

Kafka - AWS MSK 2.2.1
fluentd 1.11.0
I'm running fluentd in container based on fluent/fluentd:v1.11.0-debian-1.0 with additional plugins enabled.

gem 'fluent-plugin-kafka' version '0.13.0'

<source>
  @type kafka_group
  @id kafka_group_input
  brokers ...,...,...
  consumer_group fluentd1
  topics fluentd1
  format json
  ssl_ca_certs_from_system
</source>

# Error in logs with further shorter retries
2020-06-21 05:10:40 -0400 [error]: #0 [kafka_group_input] unexpected error during consuming events from kafka. Re-fetch events. error="undefined method `try' for #<Kafka::FetchedMessage:0x00007f3cd78c3e38>"
  2020-06-21 05:10:40 -0400 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:319:in `block (2 levels) in each_batch'
  2020-06-21 05:10:40 -0400 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:310:in `each'
  2020-06-21 05:10:40 -0400 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:310:in `block in each_batch'
  2020-06-21 05:10:40 -0400 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:414:in `block in consumer_loop'
  2020-06-21 05:10:40 -0400 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.1.0/lib/kafka/instrumenter.rb:23:in `instrument'
  2020-06-21 05:10:40 -0400 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.1.0/lib/kafka/instrumenter.rb:35:in `instrument'
  2020-06-21 05:10:40 -0400 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:412:in `consumer_loop'
  2020-06-21 05:10:40 -0400 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.1.0/lib/kafka/consumer.rb:307:in `each_batch'
  2020-06-21 05:10:40 -0400 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.13.0/lib/fluent/plugin/in_kafka_group.rb:230:in `run'
2020-06-21 05:10:40 -0400 [warn]: #0 [kafka_group_input] Stopping Consumer
2020-06-21 05:10:40 -0400 [warn]: #0 [kafka_group_input] Could not connect to broker. retry_time:1. Next retry will be in 30 seconds
2020-06-21 05:10:40.114046663 -0400 fluent.error: {"error":"undefined method `try' for #<Kafka::FetchedMessage:0x00007f3cd78c3e38>","message":"[kafka_group_input] unexpected error during consuming events from kafka. Re-fetch events. error=\"undefined method `try' for #<Kafka::FetchedMessage:0x00007f3cd78c3e38>\""}
2020-06-21 05:10:40.114562718 -0400 fluent.warn: {"message":"[kafka_group_input] Stopping Consumer"}
2020-06-21 05:10:40.114594685 -0400 fluent.warn: {"message":"[kafka_group_input] Could not connect to broker. retry_time:1. Next retry will be in 30 seconds"}

This is how I create topic:

bin/kafka-topics.sh --zookeeper ...,...,... --create --replication-factor 2 --partitions 10--topic fluentd1

Switching back to single consumer... or even to topic with just one partition works fine.. but my central logging setup really struggles.

[ncepuwanghui]

I encountered the same problem #3064
@den-is How did you solve it, is there a solution. Thanks.

[den-is]

forgot to leave a comment.
I was not able to fix it via the docker way.
But I was able to make it running on a plain CentOS Linux setup running official td-agent

Today I've tried to revisit the docker setup and review it.
Basically here is my all-in-one-basic-default-shiny fluentd image:

FROM fluent/fluentd:v1.11.1-debian-1.0

USER root

RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev" \
  && apt-get -y update \
  && apt-get -y install --no-install-recommends \
      $buildDeps \
      tzdata \
      libgeoip-dev \
      libmaxminddb-dev \
      libcurl4-openssl-dev \
  # fluentd plugins
  && fluent-gem install --no-document \
        # input/output plugins
        fluent-plugin-elasticsearch \
        fluent-plugin-record-reformer \
        fluent-plugin-kafka \
        fluent-plugin-prometheus \
        fluent-plugin-systemd \
        fluent-plugin-rewrite \
        # filter plugins
        fluent-plugin-rewrite-tag-filter \
        fluent-plugin-record-modifier \
        fluent-plugin-geoip \
        fluent-plugin-concat \
        fluent-plugin-route \
        # parser plugins
        fluent-plugin-grok-parser \
        # extra
        typhoeus \
  # cleanup
  && fluent-gem sources --clear-all \
  && SUDO_FORCE_REMOVE=yes apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false sudo \
  && apt-get clean all \
  && apt-get autoremove -y \
  && rm -rf /var/lib/apt/lists/* \
            /usr/lib/ruby/gems/*/cache/*.gem \
            /tmp/* /var/tmp/*

COPY fluent.conf /fluentd/etc/

USER fluent

Until today I had such cleanup line in the docker file:

SUDO_FORCE_REMOVE=yes apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $buildDeps \

So I kinda was removing all of that buildDeps="sudo make gcc g++ libc-dev ruby-dev" which was uninstalling quite a few other dev libraries. I hopped that I've found the issue of missing some XYZ library.
Still no luck. Nothing has worked, same old issue.

Only dependencies required on CentOS, in the above my case were: GeoIP-devel libmaxminddb-devel gcc libcurl-devel
Same plugins! Same config file. All worked great on CentOS with td-agent.

Hopefully, authors or someone else will be able to figure out what's wrong with Kafka flugin using official image. If something is missing.

[ncepuwanghui]

@den-is Okay, I try td-agent. Thank you very much.

[exherb]

related issue: zendesk/ruby-kafka#836

[micwiec]

This can be workarounded by installing first ruby-kafka from git repo.

 RUN cd /tmp && \  
     git clone https://github.com/zendesk/ruby-kafka && \  
     cd ruby-kafka && \  
     gem build && \  
     gem install *.gem

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[kenhys]

This issue was solved by fluent-plugin-kafka 0.14.0 or later.

In this version, it was changed to require ruby-kafka v1.2.0 or later and v1.2.0 contains a fix for it zendesk/ruby-kafka#841
That is why #346 (comment) works as a workaround.