Merge pull request #35 from liu-cong/envoy

Some minor fixes in Envoy setup

Author: k8s-ci-robot

pkg/README.md

@@ -10,7 +10,7 @@
 
    Our custom LLM Gateway ext-proc is patched into the existing envoy gateway via `EnvoyPatchPolicy`. To enable this feature, we must extend the Envoy Gateway config map. To do this, simply run:
    ```bash
-   kubectl apply -f ./manifests/gateway/enable_patch_policy.yaml
+   kubectl apply -f ./manifests/enable_patch_policy.yaml
    kubectl rollout restart deployment envoy-gateway -n envoy-gateway-system
 
    ```
@@ -20,14 +20,14 @@
 1. **Deploy Gateway**
 
    ```bash
-   kubectl apply -f ./manifests/gateway/gateway.yaml
+   kubectl apply -f ./manifests/gateway.yaml
    ```
 
 1. **Deploy Ext-Proc**
 
    ```bash
-   kubectl apply -f ./manifests/gateway/ext_proc.yaml
-   kubectl apply -f ./manifests/gateway/patch_policy.yaml
+   kubectl apply -f ./manifests/ext_proc.yaml
+   kubectl apply -f ./manifests/patch_policy.yaml
    ```
    **NOTE**: Ensure the `instance-gateway-ext-proc` deployment is updated with the pod names and internal IP addresses of the vLLM replicas. This step is crucial for the correct routing of requests based on headers. This won't be needed once we make ext proc dynamically read the pods.
 

pkg/manifests/ext_proc.yaml

@@ -17,15 +17,14 @@ spec:
     spec:
       containers:
       - name: instance-gateway-ext-proc
+        # TODO(https://github.com/kubernetes-sigs/llm-instance-gateway/issues/34) Update the image and args.
         image: <BUILT-IMAGE>
         args:
-        #TODO: specify label selector and dynamically update pods
+        # TODO(https://github.com/kubernetes-sigs/llm-instance-gateway/issues/12) Remove this once ext proc can dynamically reconcile on LLMServerPool.
         - -pods
         - "vllm-78665f78c4-h4kx4,vllm-78665f78c4-hnz84"
         - -podIPs
         - "10.24.11.6:8000,10.24.5.7:8000"
-        - -enable-fairness
-        - "false"
         ports:
         - containerPort: 9002
       - name: curl

pkg/manifests/gateway.yaml

@@ -38,7 +38,7 @@ metadata:
   name: llm-route
 spec:
   parentRefs:
-    - name: inference-gateway
+    - name: <GATEWAY-NAME>
       sectionName: llm-gw
   rules:
   - backendRefs:

pkg/manifests/patch_policy.yaml

@@ -34,5 +34,5 @@ spec:
       name: default/<GATEWAY-NAME>/llm-gw
       operation:
         op: replace
-        path: "/virtual_hosts/1/routes/0/route/cluster"
+        path: "/virtual_hosts/0/routes/0/route/cluster"
         value: original_destination_cluster