Document model server compatibility and config options

[liu-cong]

Fixes #482

Part of #523

Tested updated helm chart:

 helm template epp config/charts/inferencepool/ --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct --set provider.name=gke --set inferencePool.modelServerType=triton-tensorrt-llm

And the output yaml contained the Triton specific metrics:

containers:
      - name: epp
        image: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:main
        imagePullPolicy: Always
        args:
        - -poolName
        - epp
        - -poolNamespace
        - default
        - -v
        - "3"
        - -grpcPort
        - "9002"
        - -grpcHealthPort
        - "9003"
        - -metricsPort
        - "9090"
        - -totalQueuedRequestsMetric
        - "nv_trt_llm_request_metrics{request_type=waiting}"
        - -kvCacheUsagePercentageMetric
        - "nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type=fraction}"

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	fc65688
🔍 Latest deploy log	https://app.netlify.com/sites/gateway-api-inference-extension/deploys/67f94f05f17555000870fbdb
😎 Deploy Preview	https://deploy-preview-537--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[ahg-g]

Thanks

[liu-cong]

/hold I will wait for the Triton metric PR to be merged.

[kfswain]

/retest

[kfswain]

I will wait for the Triton metric PR to be merged.

Will you link that PR here so there is a paper trail?

[liu-cong]

Will you link that PR here so there is a paper trail?

This is already available in the latest Triton release. Just updated the PR to reflect that. This is ready to go! @kfswain

[liu-cong]

@ahg-g The latest Triton release is out, I updated. This is ready to be merged. Pls review. Thanks!

@BenjaminBraunDev Pls verify the Triton specific metrics configuration is correct.

[BenjaminBraunDev]

@liu-cong One flag you might want to add below the totalQueuedRequestsMetric and kvCacheUsagePercentageMetric is an empty loraInfoMetric to tell the metrics scraper not to attempt to scrap LoRA metrics for triton since they won't be present:

- -loraInfoMetric
- ""

[liu-cong]

@liu-cong One flag you might want to add below the totalQueuedRequestsMetric and kvCacheUsagePercentageMetric is an empty loraInfoMetric to tell the metrics scraper not to attempt to scrap LoRA metrics for triton since they won't be present:

- -loraInfoMetric
- ""

Great suggestion! Added

[ahg-g]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, liu-cong

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ahg-g]

/hold cancel