Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instrument create_time from last message in batch #811

Merged
merged 1 commit into from
Mar 13, 2020

Conversation

jonmagic
Copy link
Contributor

Why?

My team at GitHub uses racecar version 1.0.1 and by extension ruby-kafka. We're interested in the potential of racecar 2 that uses the rdkafka-ruby client but we heavily rely on the consumer.offset_lag metric for monitoring our stream processors.

It doesn't look like that metric is going to be straightforward to implement in racecar 2 but I'm wondering if we could instead switch to the consumer.time_lag metric instead? Unfortunately that metric isn't published in the process_batch.consumer payload in racecar 1.0.1 or racecar 2.0.0-beta.

How?

This PR updates the process_batch.consumer payload to include a last_create_time metric which comes from the last message in the batch.

We could use a fork of ruby-kafka with these changes to accomplish our goal with racecar 1.0.1 in the short term but I thought others may benefit as well so I'm opening this PR.

I'm also hoping to open a PR against racecar 2 to add the same metric to the process_batch payload so we can easily transition from racecar 1.0.1 with ruby-kafka to racecar 2.0.0 with rdkafka-ruby.

Thank you for your consideration.

@jonmagic
Copy link
Contributor Author

The failing specs should all pass once #810 ships and I update this PR to include it.

@dasch
Copy link
Contributor

dasch commented Feb 23, 2020

Can you rebase on master?

@dasch dasch merged commit 1b2ed7c into zendesk:master Mar 13, 2020
@dasch
Copy link
Contributor

dasch commented Mar 13, 2020

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants