Instrument create_time from last message in batch #811
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why?
My team at GitHub uses racecar version 1.0.1 and by extension ruby-kafka. We're interested in the potential of racecar 2 that uses the rdkafka-ruby client but we heavily rely on the
consumer.offset_lag
metric for monitoring our stream processors.It doesn't look like that metric is going to be straightforward to implement in racecar 2 but I'm wondering if we could instead switch to the
consumer.time_lag
metric instead? Unfortunately that metric isn't published in theprocess_batch.consumer
payload in racecar 1.0.1 or racecar 2.0.0-beta.How?
This PR updates the
process_batch.consumer
payload to include alast_create_time
metric which comes from the last message in the batch.We could use a fork of
ruby-kafka
with these changes to accomplish our goal with racecar 1.0.1 in the short term but I thought others may benefit as well so I'm opening this PR.I'm also hoping to open a PR against racecar 2 to add the same metric to the
process_batch
payload so we can easily transition from racecar 1.0.1 with ruby-kafka to racecar 2.0.0 with rdkafka-ruby.Thank you for your consideration.