diff --git a/docs/endpoint-picker.svg b/docs/endpoint-picker.svg new file mode 100644 index 00000000..3ec8eed4 --- /dev/null +++ b/docs/endpoint-picker.svg @@ -0,0 +1,3 @@ +Endpoint PickerServiceModelServerL7 Proxy / Gateway InferencePool API Selects - the model servers (the endpoints) - the endpoint picker serviceModel ServerProtocolTrafficExtensionProtocolGateway ControllerClientTrafficConfiguresWatchesWatches InferenceModel API Defines - the model/adapter to serve - the serving objectives for the modelObservabilityMetrics ScrapingObservabilityDashboardsStandard GatewayElementsInference ExtensionElementsInference Gateway \ No newline at end of file diff --git a/pkg/epp/README.md b/pkg/epp/README.md index e3bc26ae..1bf47993 100644 --- a/pkg/epp/README.md +++ b/pkg/epp/README.md @@ -1,8 +1,12 @@ # The EndPoint Picker (EPP) -This package provides the reference implementation for the Endpoint Picker (EPP). It implements the [extension protocol](../../docs/proposals/003-endpoint-picker-protocol), enabling a proxy or gateway to request endpoint hints from an extension. An EPP instance handles a single `InferencePool` (and so for each `InferencePool`, one must create a dedicated EPP deployment). +This package provides the reference implementation for the Endpoint Picker (EPP). As demonistrated in the diagram below, it implements the [extension protocol](../../docs/proposals/004-endpoint-picker-protocol), enabling a proxy or gateway to request endpoint hints from an extension, and interacts with the model servers through the defined [model server protocol](../..//docs/proposals/003-model-server-protocol). +![Architecture Diagram](../../docs/endpoint-picker.svg) -The Endpoint Picker performs the following core functions: + +## Core Functions + +An EPP instance handles a single `InferencePool` (and so for each `InferencePool`, one must create a dedicated EPP deployment), it performs the following core functions: - Endpoint Selection - The EPP determines the appropriate Pod endpoint for the load balancer (LB) to route requests. @@ -17,8 +21,8 @@ The Endpoint Picker performs the following core functions: - It reports InferenceModel-level metrics, further broken down by target model. - Detailed information regarding metrics can be found on the [website](https://gateway-api-inference-extension.sigs.k8s.io/guides/metrics/). -## The scheduling algorithm + +## Scheduling Algorithm The scheduling package implements request scheduling algorithms for load balancing requests across backend pods in an inference gateway. The scheduler ensures efficient resource utilization while maintaining low latency and prioritizing critical requests. It applies a series of filters based on metrics and heuristics to select the best pod for a given request. The following flow chart summarizes the current scheduling algorithm -# Flowchart Scheduling Algorithm