Adding getting started instructions for GKE, Istio, and Kgateway

[nicolexin]

Update inference extension getting started guide:

• Add instructions for GKE, Istio and Kgateway
• Remove instructions for Envoy Gateway until they have controller-level support

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: nicolexin / name: Nicole Xin (9343660, 0519935, 3e7e74e, 6d48b5b, a140a3e, 8d235f6, d8d4666, 557c44f, 5a2677e, 63d7c40, 52318b3, 8ef12a8, 7b490de, ee7fa97, efb8c35, 8a878f8, 21100f9, e82e074, a627ea7, 9c8d00d, f6f9538, f0b59e4, 59cbe2e, afc64dc, 048189a, 35a835f, e512145, c06cffd, ff8b2a1, 6bc07c6, d493258, 2f9baea, c1b563b, 484f19f, 9cb2575, 6a9f91a, 0a24389, 2574453, 365d847, e4471ec, ce19438, e9f2298, c82487d, a679070, d0ddd16, b63263d, e1c0b1d, b6d4c7a, 6d3642a, d71f29c, d5fd70f, 41fc083)

[k8s-ci-robot]

Welcome @nicolexin!

It looks like this is your first PR to kubernetes-sigs/gateway-api-inference-extension 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gateway-api-inference-extension has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @nicolexin. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	c82487d
🔍 Latest deploy log	https://app.netlify.com/sites/gateway-api-inference-extension/deploys/67e71fbefa645e00082a1796
😎 Deploy Preview	https://deploy-preview-577--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[liu-cong]

/ok-to-test

[robscott]

Thanks @nicolexin!

For KGateway:
/cc @danehans @christian-posta

For Istio:
/cc @LiorLieberman

[robscott]

Thanks @nicolexin!

[robscott]

@LiorLieberman My goal with the suggestion was to highlight that Istio's TLS verification is a positive/helpful feature. IMO, the only time neutral OSS docs should point out a shortcoming of an implementation is if that implementation is failing to do something required by the API, that's not the case here.

[LiorLieberman]

Thanks for all the work here @nicolexin! overall LGTM.
minor nits on some threads (nothing is actionable for now probably)

/lgtm

[robscott]

Thanks @nicolexin!

/lgtm

[nicolexin]

/assign kfswain

[ahg-g]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, LiorLieberman, nicolexin, robscott

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nicolexin]

I've added my review of the instructions for kgateway and updated the thread above with the steps/commands.

A few thoughts from me:

1. the gpu-deployment uses "vllm/vllm-openai:latest"; is that intended? I think we should pin to a specific version.

https://github.com/nicolexin/gateway-api-inference-extension/blob/userguide/config/manifests/vllm/gpu-deployment.yaml#L17

2. this ClusterRoleBinding gives me an error when I apply it:

https://github.com/nicolexin/gateway-api-inference-extension/blob/userguide/config/manifests/inferencepool.yaml#L117

inferencepool.inference.networking.x-k8s.io/vllm-llama3-8b-instruct created
service/vllm-llama3-8b-instruct-epp created
deployment.apps/vllm-llama3-8b-instruct-epp created
clusterrole.rbac.authorization.k8s.io/pod-read unchanged
error: error validating "https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencepool.yaml": error validating data: ValidationError(ClusterRoleBinding.roleRef): missing required field "apiGroup" in io.k8s.api.rbac.v1.RoleRef; if you choose to ignore these errors, turn validation off with --validate=false

3. With the startupProbe and agressive liveness/readiness I could not get the llm container to come up FYI. I understand why these settings are important -- we may need to fiddle with them a little to arrive at good defaults. Have others had any issues with this? I've had to back those out on my side to get the llms to work...

Thanks!

I did run through the guide with a GKE cluster end to end and I have no issues applying the ClusterRoleBinding and the vLLM deployments.
@LiorLieberman - did you run into any of the issues above?

[kfswain]

@christian-posta

1. the gpu-deployment uses "vllm/vllm-openai:latest"; is that intended? I think we should pin to a specific version.

Yeah, we mark a specific branch in our version branches. Granted that doesn't make it to our site (we only host main). We may need to break out version specific guides. Cut: #610

2. this ClusterRoleBinding gives me an error when I apply it:

Interesting, I think that's just a validation error, and they RoleBinding should still exist afaik? I did omit apiGroup but I think it defaults to rbac.authorization.k8s.io/v1. I always err on the side of less config, but we can fix this if it full errors out.

3. With the startupProbe and agressive liveness/readiness I could not get the llm container to come up FYI. I understand why these settings are important -- we may need to fiddle with them a little to arrive at good defaults. Have others had any issues with this? I've had to back those out on my side to get the llms to work...

They work for me. I'm using A100s its possible we need to have a disclaimer that they are tuned for A100 machines. LMK

[kfswain]

Thanks @nicolexin!!! RIP xDS Surgery, you won't be missed :P