Introduce SaturationDetector component #808
Conversation
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
k8s-ci-robot
added
cncf-cla: yes
|
Hi @LukeAVanDrie. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/XL
LukeAVanDrie
force-pushed
the
saturation-detector-impl
branch
from
f10f62f to
37e73c4
Compare
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
LukeAVanDrie
force-pushed
the
saturation-detector-impl
branch
2 times, most recently
from
be326b9 to
5217359
Compare
|
/retest |
|
/approve Will wait for tests/ response to comments before LGTM. Thanks Luke! |
k8s-ci-robot
added
the
approved
Test failure is from an unrelated flake. I sent out a fix in #824 |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
This commit adds a new `SaturationDetector` component responsible for determining if backend model servers are saturated. It bases its decision on observed metrics like queue depth and KV cache utilization, using configurable thresholds. The detector is designed to be a self-contained unit that can be leveraged by other components for admission control and capacity assessment. This is the first step in a larger refactoring to externalize and centralize saturation detection logic.
LukeAVanDrie
force-pushed
the
saturation-detector-impl
branch
from
5217359 to
33342b6
Compare
k8s-ci-robot
removed
the
lgtm
I cannot keep up with @nirrozenbaum :) Rebased on to the MetricsState change which should fix the failing tests. |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kfswain, LukeAVanDrie The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This commit adds a new `SaturationDetector` component responsible for determining if backend model servers are saturated. It bases its decision on observed metrics like queue depth and KV cache utilization, using configurable thresholds. The detector is designed to be a self-contained unit that can be leveraged by other components for admission control and capacity assessment. This is the first step in a larger refactoring to externalize and centralize saturation detection logic.
This commit adds a new `SaturationDetector` component responsible for determining if backend model servers are saturated. It bases its decision on observed metrics like queue depth and KV cache utilization, using configurable thresholds. The detector is designed to be a self-contained unit that can be leveraged by other components for admission control and capacity assessment. This is the first step in a larger refactoring to externalize and centralize saturation detection logic.
This commit adds a new
SaturationDetectorcomponent responsible for determining if backend model servers are saturated. It bases its decision on observed metrics like queue depth and KV cache utilization, using configurable thresholds.To be submitted before #805 (split this change out for easier review).
Related to: #674
It's initial implementation uses identical thresholds to the
HasCapacityfilter embedded in the default Scheduler filter chain for sheddable requests.The detector is designed to be a self-contained unit that can be leveraged by other components for admission control and capacity assessment. This is the first step in a larger refactoring to externalize and centralize saturation detection logic.