You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A chart to deploy the inference extension and a InferencePool managed by the extension.
3
+
A chart to deploy an InferencePool and a corresponding EndpointPicker (epp) deployment.
4
4
5
-
## Install
6
5
7
-
Suppose now a vllm service with label `app: vllm-llama2-7b` and served on port `8000` is deployed in `default` namespace in the cluster.
6
+
## Install
8
7
9
-
To deploy the inference extension, you can run the following command:
8
+
To install an InferencePool named `pool-1` that selects from endpoints with label `app: vllm-llama2-7b` and listening on port `8000`, you can run the following command:
where `inferencePool.targetPortNumber` is the pod that vllm backends served on and `inferencePool.selector` is the selector to match the vllm backends. And then run:
28
-
29
-
```txt
30
-
$ helm install my-release .
31
-
```
17
+
where `inferencePool.targetPortNumber` is the pod that vllm backends served on and `inferencePool.selector` is the selector to match the vllm backends.
32
18
33
19
## Uninstall
34
20
@@ -44,18 +30,16 @@ The following table list the configurable parameters of the chart.
|`inferencePool.name`| Name for the InferencePool, and inference extension will be named as `${inferencePool.name}-epp`. |
34
+
|`inferencePool.targetPortNumber`| Target port number for the vllm backends, will be used to scrape metrics by the inference extension. |
35
+
|`inferencePool.selector`| Label selector to match vllm backends managed by the inference pool. |
47
36
|`inferenceExtension.replicas`| Number of replicas for the inference extension service. Defaults to `1`. |
48
37
|`inferenceExtension.image.name`| Name of the container image used for the inference extension. |
49
38
|`inferenceExtension.image.hub`| Registry URL where the inference extension image is hosted. |
50
39
|`inferenceExtension.image.tag`| Image tag of the inference extension. |
51
40
|`inferenceExtension.image.pullPolicy`| Image pull policy for the container. Possible values: `Always`, `IfNotPresent`, or `Never`. Defaults to `Always`. |
52
41
|`inferenceExtension.extProcPort`| Port where the inference extension service is served for external processing. Defaults to `9002`. |
53
-
| `inferencePool.name` | Name for the InferencePool, and inference extension will be named as `${inferencePool.name}-epp`. |
54
-
| `inferencePool.targetPortNumber` | Target port number for the vllm backends, will be used to scrape metrics by the inference extension. |
55
-
| `inferencePool.selector` | Label selector to match vllm backends managed by the inference pool. |
56
42
57
43
## Notes
58
44
59
-
This chart will only deploy the inference extension and InferencePool, before install the chart, please make sure that the inference extension CRDs have already been installed in the cluster. And You need to apply traffic policies to route traffic to the inference extension from the gateway after the inference extension is deployed.
60
-
61
-
For more details, please refer to the [website](https://gateway-api-inference-extension.sigs.k8s.io/guides/).
45
+
This chart will only deploy an InferencePool and its corresponding EndpointPicker extension. Before install the chart, please make sure that the inference extension CRDs are installed in the cluster. For more details, please refer to the [getting started guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/).
0 commit comments