Skip to content

Latest commit

 

History

History
66 lines (46 loc) · 3.52 KB

README.md

File metadata and controls

66 lines (46 loc) · 3.52 KB

Endpoint Picker Protocol

The Endpoint Picker, or EPP, is a core component of the inference extension. Ultimately it's responsible for picking an endpoint from the InferencePool. A reference implementation can be found here.

Proxy Protocol

This is the protocol between the EPP and the proxy (e.g, Envoy).

The EPP MUST implement the Envoy external processing serviceprotocol.

For each HTTP request, the EPP MUST communicate to the proxy the picked model server endpoint, via adding the x-gateway-destination-endpoint HTTP header in the request and as an unstructured entry in the dynamic_metadata field of the ext-proc response, or otherwise return an error. The EPP MUST not set two different values in the header and the response metadata. Setting different value leads to unpredictable behavior because proxies aren't guaranteed to support both paths, and so this protocol does not define what takes precedence.

Model Server Protocol

This is the protocol between the EPP and the model servers.

Inference API Protocol

The model server MUST implement OpenAI’s Completions and Chat APIs.

Metrics Reporting

The inference extension scrapes metrics from the model servers to make optimal request scheduling decisions. The model servers MUST provide the following metrics via a Prometheus endpoint. The exact metric names don't necessarily need to be the same as the recommended names here, however the metric types and semantics MUST follow this doc.

Note the requirements here are aligned with the model server metrics standardization effort.

The corresponding metrics in vLLM are also shown in the table below, as vLLM is already integrated into the reference endpoint picker implementation.

Metric Type Description vLLM metric
TotalQueuedRequests Gauge The current total number of requests in the queue. vllm:num_requests_waiting
KVCacheUtilization Gauge The current KV cache utilization in percentage. vllm:gpu_cache_usage_perc

LoRA Adapter Serving

Model servers that support dynamic LoRA serving can benefit from the LoRA affinity algorithm. Note the current algorithm in the reference EPP is highly biased towards vLLM's current dynamic LoRA implementation.

The model servers MUST support serving a LoRA adapter specified in the model argument of the request, provided the requested adapter is valid.

The model server MUST expose the following LoRA adapter metrics via the same Prometheus endpoint:

  • Metric name implemented in vLLM: vllm:lora_requests_info
  • Metric type: Gauge
  • Metric value: The last updated timestamp (so the EPP can find the latest).
  • Metric labels:
    • max_lora: The maximum number of adapters that can be loaded to GPU memory to serve a batch. Requests will be queued if the model server has reached MaxActiveAdapter and canno load the requested adapter. Example: "max_lora": "8".
    • running_lora_adapters: A comma separated list of adapters that are currently loaded in GPU memory and ready to serve requests. Example: "running_lora_adapters": "adapter1, adapter2"