Add running request gauge metric #604
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
k8s-ci-robot
added
the
size/M
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
pkg/epp/handlers/streamingserver.go
Outdated
| @@ -322,6 +323,7 @@ func (s *StreamingServer) HandleRequestBody( | |||
| if !ok { | |||
| return reqCtx, errutil.Error{Code: errutil.BadRequest, Msg: "model not found in request"} | |||
| } | |||
| metrics.IncRunningRequests(model) | |||
There was a problem hiding this comment.
Do we want to increment this only if we successfully responded with a picked endpoint? @smarterclayton wdyt?
There was a problem hiding this comment.
If an endpoint is not picked, the gauge will decrease immediately in the error handling so I think it should be fine to increase the request number it here.
k8s-ci-robot
added
size/L
|
/approve |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ahg-g, JeffLuoo The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
Tested locally and compare the result with vLLM requests count gauge metrics:
Blue Line: vLLM
Green Line: EPP
Resolve #593