Simulation code for llm inference gateway

[kaushikmitr]

The notebook implements the following components:

1. Prefill and Decode Latency Modeling as a Function of Batch Sizes:

   • The prefill and decode latencies are modeled based on batch size, where the total latency is influenced by the number of tokens in a batch.
   • The model incorporates different latency constants, providing a mathematical representation of how batch sizes impact the prefill and decode processes.
   • Latency is calculated using a formula that sums up token lengths across the batch, giving insight into the relationship between batch size and latency performance.

2. Continuous Batching Algorithm Simulation:

   • This component simulates a continuous batching mechanism where scheduling decisions are made dynamically before each forward pass as implemented in vLLM.

3. LLM Inference Gateway Simulation with Different Routing Strategies:

   • The simulation models an LLM inference gateway, where requests are routed to different model servers based on various strategies.
   • Multiple routing algorithms are explored to optimize how requests are handled, balancing latency, throughput, and resource utilization.

4. Evaluating Different Algorithms with Configurable Workloads:

   • The notebook enables the evaluation of various algorithms under different configurable workloads, allowing for flexible testing of different conditions.
   • Users can configure workloads by modifying factors such as request sizes, batch sizes, token lengths, and latency requirements.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: kaushikmitr (33f20a9, d793532, 9bc831c, cf7d0b3, a69db6c, 8e94a9c, 58d2c77, a6ee825, b9d12e3, 1fdb7b5, 1e9e3cc, 5d48994)

[k8s-ci-robot]

Welcome @kaushikmitr!

It looks like this is your first PR to kubernetes-sigs/llm-instance-gateway 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/llm-instance-gateway has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[ahg-g]

any idea to make this reviewable on github? the file is way too big with image blobs taking up so much space

[kaushikmitr]

any idea to make this reviewable on github? the file is way too big with image blobs taking up so much space

Yes, this is more for viewing (https://github.com/kubernetes-sigs/llm-instance-gateway/blob/a6ee825c4a91d4a5c7915d82b6a6991123347c7b/simulations/llm_inference_gateway_simulation.ipynb) but hard to comment on. I am working on making this into a regular python script.

[kaushikmitr]

added python library llm_ig_simulation folder

[terrytangyuan]

Remove extra lines (here and other places)

[kaushikmitr]

done

[ahg-g]

When you get a chance, can we have readme file describing how to run this?

[ahg-g]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, kaushikmitr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment