From c62d3b3026abef07cca4a91d38572020ffd17088 Mon Sep 17 00:00:00 2001 From: "wangbaiping(wbpcode)" Date: Wed, 30 Apr 2025 21:34:22 +0800 Subject: [PATCH 1/5] Amend the endpoint picker protocol to support multiple fallback endpoints Signed-off-by: wangbaiping(wbpcode) --- .../004-endpoint-picker-protocol/README.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/proposals/004-endpoint-picker-protocol/README.md b/docs/proposals/004-endpoint-picker-protocol/README.md index 5280e05cb..51e97eed7 100644 --- a/docs/proposals/004-endpoint-picker-protocol/README.md +++ b/docs/proposals/004-endpoint-picker-protocol/README.md @@ -44,11 +44,23 @@ Constraints: - If the EPP did not communicate the server endpoint via these two methods, it MUST return an error as follows: - [ImmediateResponse](https://github.com/envoyproxy/envoy/blob/f2023ef77bdb4abaf9feef963c9a0c291f55568f/api/envoy/service/ext_proc/v3/external_processor.proto#L195) with 503 (Serivce Unavailable) HTTP status code if there are no ready endpoints. - [ImmediateResponse](https://github.com/envoyproxy/envoy/blob/f2023ef77bdb4abaf9feef963c9a0c291f55568f/api/envoy/service/ext_proc/v3/external_processor.proto#L195) with 429 (Too Many Requests) HTTP status code if the request should be dropped (e.g., a Sheddable request, and the servers under heavy load). -- The EPP MUST not set two different values in the header and the inner response metadata value. +- The EPP MUST not set two different values in the header and the inner response metadata value. - Setting different value leads to unpredictable behavior because proxies aren't guaranteed to support both paths, and so this protocol does not define what takes precedence. ### Destination endpoint fallback -A single fallback endpoint CAN be set using the key `x-gateway-destination-endpoint-fallback` in the same metadata namespace as one used for `x-gateway-destination-endpoint` as follows: + +For each HTTP request, if destination endpoint fallback is necessary or possible, the EPP CAN set the `x-gateway-destination-endpoint` HTTP header or metadata entry with multiple addresses in `,,...` format. Multiple addresses are separated by commas. The first valid endpoint in the addresses list will be used as the primary endpoint. And if retrying is happening, the proxy will try the endpoints after the selected endpoint in order. + +For example: +```go +dynamicMetadata: { + "envoy.lb" { + "x-gateway-destination-endpoint": ",,..." + } +} +``` + +Single fallback endpoint also CAN be set using the key `x-gateway-destination-endpoint-fallback` in the same metadata namespace as one used for `x-gateway-destination-endpoint` as follows: ```go dynamicMetadata: { @@ -58,7 +70,7 @@ dynamicMetadata: { } ``` -### Why envoy.lb namespace as a default? +### Why envoy.lb namespace as a default? The `envoy.lb` namespace is a predefined namespace. One common way to use the selected endpoint returned from the server, is [envoy subsets](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/subsets) where host metadata for subset load balancing must be placed under `envoy.lb`. Note that this is not related to the subsetting feature discussed above, this is an enovy implementation detail. ## Matching An InferenceModel From 1047cb6c6f87e956c67ff3dd4cc1ae57ad3a97c5 Mon Sep 17 00:00:00 2001 From: "wangbaiping(wbpcode)" Date: Fri, 9 May 2025 11:24:41 +0800 Subject: [PATCH 2/5] address comments Signed-off-by: wangbaiping(wbpcode) --- .../004-endpoint-picker-protocol/README.md | 35 +++++++++---------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/docs/proposals/004-endpoint-picker-protocol/README.md b/docs/proposals/004-endpoint-picker-protocol/README.md index 51e97eed7..8e96214e9 100644 --- a/docs/proposals/004-endpoint-picker-protocol/README.md +++ b/docs/proposals/004-endpoint-picker-protocol/README.md @@ -27,9 +27,9 @@ If the key `x-gateway-destination-endpoint-subset` is not set, then the EPP MUST ## Destination Endpoint For each HTTP request, the EPP MUST communicate to the proxy the picked model server endpoint via: -1. Setting the `x-gateway-destination-endpoint` HTTP header to the selected endpoint in format. +1. Setting the `x-gateway-destination-endpoint` HTTP header to the selected endpoints. -2. Set an unstructured entry in the [dynamic_metadata](https://github.com/envoyproxy/go-control-plane/blob/c19bf63a811c90bf9e02f8e0dc1dcef94931ebb4/envoy/service/ext_proc/v3/external_processor.pb.go#L320) field of the ext-proc response. The metadata entry for the picked endpoint MUST be wrapped with an outer key (which represents the metadata namespace) with a default of `envoy.lb`. +2. Set an unstructured entry in the [dynamic_metadata](https://github.com/envoyproxy/go-control-plane/blob/c19bf63a811c90bf9e02f8e0dc1dcef94931ebb4/envoy/service/ext_proc/v3/external_processor.pb.go#L320) field of the ext-proc response. The metadata entry for the picked endpoints MUST be wrapped with an outer key (which represents the metadata namespace) with a default of `envoy.lb`. The primary endpoint MUST be set using the key `x-gateway-destination-endpoint` as follows: ```go @@ -39,28 +39,18 @@ dynamicMetadata: { } } ``` - -Constraints: -- If the EPP did not communicate the server endpoint via these two methods, it MUST return an error as follows: - - [ImmediateResponse](https://github.com/envoyproxy/envoy/blob/f2023ef77bdb4abaf9feef963c9a0c291f55568f/api/envoy/service/ext_proc/v3/external_processor.proto#L195) with 503 (Serivce Unavailable) HTTP status code if there are no ready endpoints. - - [ImmediateResponse](https://github.com/envoyproxy/envoy/blob/f2023ef77bdb4abaf9feef963c9a0c291f55568f/api/envoy/service/ext_proc/v3/external_processor.proto#L195) with 429 (Too Many Requests) HTTP status code if the request should be dropped (e.g., a Sheddable request, and the servers under heavy load). -- The EPP MUST not set two different values in the header and the inner response metadata value. -- Setting different value leads to unpredictable behavior because proxies aren't guaranteed to support both paths, and so this protocol does not define what takes precedence. - -### Destination endpoint fallback - -For each HTTP request, if destination endpoint fallback is necessary or possible, the EPP CAN set the `x-gateway-destination-endpoint` HTTP header or metadata entry with multiple addresses in `,,...` format. Multiple addresses are separated by commas. The first valid endpoint in the addresses list will be used as the primary endpoint. And if retrying is happening, the proxy will try the endpoints after the selected endpoint in order. - -For example: +or: ```go dynamicMetadata: { - "envoy.lb" { - "x-gateway-destination-endpoint": ",,..." + "envoy.lb": { + "x-gateway-destination-endpoint": ,,... } } ``` -Single fallback endpoint also CAN be set using the key `x-gateway-destination-endpoint-fallback` in the same metadata namespace as one used for `x-gateway-destination-endpoint` as follows: +The value of the header or metadata entry MUST contains at least one endpoint in `` format or multiple endpoints in `,,...` format. Multiple endpoints are separated by commas. The first valid endpoint in the value will be used. And if retrying is happening, the proxy will try the endpoints after the previously selected endpoint in order. + +The EPP also can set additional endpoints by the key `x-gateway-destination-endpoint-fallback` in the same metadata namespace as one used for `x-gateway-destination-endpoint` as follows: ```go dynamicMetadata: { @@ -70,6 +60,15 @@ dynamicMetadata: { } ``` +The endpoints specified in `x-gateway-destination-endpoint-fallback` MAY be tried after the endpoints specified in `x-gateway-destination-endpoint` if all the endpoints specified in `x-gateway-destination-endpoint` are unavailable. + +Constraints: +- If the EPP did not communicate the server endpoint via these two methods, it MUST return an error as follows: + - [ImmediateResponse](https://github.com/envoyproxy/envoy/blob/f2023ef77bdb4abaf9feef963c9a0c291f55568f/api/envoy/service/ext_proc/v3/external_processor.proto#L195) with 503 (Serivce Unavailable) HTTP status code if there are no ready endpoints. + - [ImmediateResponse](https://github.com/envoyproxy/envoy/blob/f2023ef77bdb4abaf9feef963c9a0c291f55568f/api/envoy/service/ext_proc/v3/external_processor.proto#L195) with 429 (Too Many Requests) HTTP status code if the request should be dropped (e.g., a Sheddable request, and the servers under heavy load). +- The EPP MUST not set two different values in the header and the inner response metadata value. +- Setting different value leads to unpredictable behavior because proxies aren't guaranteed to support both paths, and so this protocol does not define what takes precedence. + ### Why envoy.lb namespace as a default? The `envoy.lb` namespace is a predefined namespace. One common way to use the selected endpoint returned from the server, is [envoy subsets](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/subsets) where host metadata for subset load balancing must be placed under `envoy.lb`. Note that this is not related to the subsetting feature discussed above, this is an enovy implementation detail. From f7d96a858aab858f99bb76703fc6c795f7b092ec Mon Sep 17 00:00:00 2001 From: "wangbaiping(wbpcode)" Date: Fri, 9 May 2025 11:29:24 +0800 Subject: [PATCH 3/5] add optionally Signed-off-by: wangbaiping(wbpcode) --- docs/proposals/004-endpoint-picker-protocol/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/004-endpoint-picker-protocol/README.md b/docs/proposals/004-endpoint-picker-protocol/README.md index 8e96214e9..6c1192ac7 100644 --- a/docs/proposals/004-endpoint-picker-protocol/README.md +++ b/docs/proposals/004-endpoint-picker-protocol/README.md @@ -50,7 +50,7 @@ dynamicMetadata: { The value of the header or metadata entry MUST contains at least one endpoint in `` format or multiple endpoints in `,,...` format. Multiple endpoints are separated by commas. The first valid endpoint in the value will be used. And if retrying is happening, the proxy will try the endpoints after the previously selected endpoint in order. -The EPP also can set additional endpoints by the key `x-gateway-destination-endpoint-fallback` in the same metadata namespace as one used for `x-gateway-destination-endpoint` as follows: +Optionally, The EPP also CAN set additional endpoints by the key `x-gateway-destination-endpoint-fallback` in the same metadata namespace as one used for `x-gateway-destination-endpoint` as follows: ```go dynamicMetadata: { From c204cf7ed3bd32760dc65dabdd3c14bca3784059 Mon Sep 17 00:00:00 2001 From: "wangbaiping(wbpcode)" Date: Tue, 13 May 2025 11:42:26 +0800 Subject: [PATCH 4/5] address comments Signed-off-by: wangbaiping(wbpcode) --- .../004-endpoint-picker-protocol/README.md | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/docs/proposals/004-endpoint-picker-protocol/README.md b/docs/proposals/004-endpoint-picker-protocol/README.md index 6c1192ac7..e89c594fe 100644 --- a/docs/proposals/004-endpoint-picker-protocol/README.md +++ b/docs/proposals/004-endpoint-picker-protocol/README.md @@ -25,9 +25,9 @@ If the key `x-gateway-destination-endpoint-subset` is set, the EPP MUST only sel If the key `x-gateway-destination-endpoint-subset` is not set, then the EPP MUST select from the set defined by the `InferencePool` selector. ## Destination Endpoint -For each HTTP request, the EPP MUST communicate to the proxy the picked model server endpoint via: +For each HTTP request, the EPP MUST communicate to the proxy one or more selected model server endpoints via: -1. Setting the `x-gateway-destination-endpoint` HTTP header to the selected endpoints. +1. Setting the `x-gateway-destination-endpoint` HTTP header to one or more selected endpoints. 2. Set an unstructured entry in the [dynamic_metadata](https://github.com/envoyproxy/go-control-plane/blob/c19bf63a811c90bf9e02f8e0dc1dcef94931ebb4/envoy/service/ext_proc/v3/external_processor.pb.go#L320) field of the ext-proc response. The metadata entry for the picked endpoints MUST be wrapped with an outer key (which represents the metadata namespace) with a default of `envoy.lb`. @@ -48,19 +48,7 @@ dynamicMetadata: { } ``` -The value of the header or metadata entry MUST contains at least one endpoint in `` format or multiple endpoints in `,,...` format. Multiple endpoints are separated by commas. The first valid endpoint in the value will be used. And if retrying is happening, the proxy will try the endpoints after the previously selected endpoint in order. - -Optionally, The EPP also CAN set additional endpoints by the key `x-gateway-destination-endpoint-fallback` in the same metadata namespace as one used for `x-gateway-destination-endpoint` as follows: - -```go -dynamicMetadata: { - "envoy.lb" { - "x-gateway-destination-endpoint-fallback": - } -} -``` - -The endpoints specified in `x-gateway-destination-endpoint-fallback` MAY be tried after the endpoints specified in `x-gateway-destination-endpoint` if all the endpoints specified in `x-gateway-destination-endpoint` are unavailable. +The value of the header or metadata entry MUST contain at least one endpoint in `` format or multiple endpoints in `,,...` format. Multiple endpoints are separated by commas. The first valid endpoint in the list will be used. And if retrying is happening, the proxy will try the endpoints after the previously selected endpoint in order. Constraints: - If the EPP did not communicate the server endpoint via these two methods, it MUST return an error as follows: From 6a3c8f699c8d1633a2542a2cddbbb314744549f1 Mon Sep 17 00:00:00 2001 From: "wangbaiping(wbpcode)" Date: Tue, 13 May 2025 11:44:48 +0800 Subject: [PATCH 5/5] address comments Signed-off-by: wangbaiping(wbpcode) --- docs/proposals/004-endpoint-picker-protocol/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/004-endpoint-picker-protocol/README.md b/docs/proposals/004-endpoint-picker-protocol/README.md index e89c594fe..9fdb67c2f 100644 --- a/docs/proposals/004-endpoint-picker-protocol/README.md +++ b/docs/proposals/004-endpoint-picker-protocol/README.md @@ -48,7 +48,7 @@ dynamicMetadata: { } ``` -The value of the header or metadata entry MUST contain at least one endpoint in `` format or multiple endpoints in `,,...` format. Multiple endpoints are separated by commas. The first valid endpoint in the list will be used. And if retrying is happening, the proxy will try the endpoints after the previously selected endpoint in order. +The value of the header or metadata entry MUST contain at least one endpoint in `` format or multiple endpoints in `,,...` format. Multiple endpoints are separated by commas. The first valid endpoint in the list will be used. If retry is configured, the proxy will go sequentially down the list until one valid endpoint is found. Constraints: - If the EPP did not communicate the server endpoint via these two methods, it MUST return an error as follows: