Lora syncer docs #320

coolkp · 2025-02-11T02:21:45Z

Fixes issue #303 .

Add a make file to lora syncer and a cloud build file to automatically create and publish the image
Update the guide to incorporate the lora syncer

k8s-ci-robot · 2025-02-11T02:21:55Z

Hi @coolkp. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2025-02-11T02:22:04Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`277125c`
🔍 Latest deploy log	https://app.netlify.com/sites/gateway-api-inference-extension/deploys/67afb18c7477390008ee1b6d
😎 Deploy Preview	https://deploy-preview-320--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

site-src/guides/index.md

ahg-g · 2025-02-12T15:44:01Z

/ok-to-test

ahg-g · 2025-02-12T15:44:46Z

I don't think you uploaded the guide

site-src/guides/dynamic-lora.md

ahg-g · 2025-02-12T20:58:01Z

pkg/manifests/vllm/deployment-with-syncer.yaml

+          - "sql-lora=/adapters/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/"
+          - "tweet-summary=/adapters/hub/models--vineetsharma--qlora-adapter-Llama-2-7b-hf-TweetSumm/snapshots/796337d8e866318c59e38f16416e3ecd11fe5403"
+          - 'sql-lora-0=/adapters/yard1/llama-2-7b-sql-lora-test_0'
+          - 'sql-lora-1=/adapters/yard1/llama-2-7b-sql-lora-test_1'
+          - 'sql-lora-2=/adapters/yard1/llama-2-7b-sql-lora-test_2'
+          - 'sql-lora-3=/adapters/yard1/llama-2-7b-sql-lora-test_3'
+          - 'sql-lora-4=/adapters/yard1/llama-2-7b-sql-lora-test_4'
+          - 'tweet-summary-0=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_0'
+          - 'tweet-summary-1=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_1'
+          - 'tweet-summary-2=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_2'
+          - 'tweet-summary-3=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_3'
+          - 'tweet-summary-4=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_4'


I would remove those and leave the two that the inferenceModel.yaml have, tweet-summary-0 and tweet-summary-1:

'tweet-summary-0=vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm'

'tweet-summary-1=vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm'

ahg-g · 2025-02-12T20:59:40Z

pkg/manifests/vllm/deployment-with-syncer.yaml

+          - base-model: meta-llama/Llama-2-7b-hf
+            id: sql-lora-v1
+            source: yard1/llama-2-7b-sql-lora-test
+          - base-model: meta-llama/Llama-2-7b-hf
+            id: sql-lora-v3
+            source: yard1/llama-2-7b-sql-lora-test
+          - base-model: meta-llama/Llama-2-7b-hf
+            id: sql-lora-v4
+            source: yard1/llama-2-7b-sql-lora-test


keep only tweet-summary-0 and 1:
- base-model: meta-llama/Llama-2-7b-hf
id: tweet-summary-0
source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm
- base-model: meta-llama/Llama-2-7b-hf
id: tweet-summary-1
source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm

need to align this with https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/manifests/inferencemodel.yaml, so only keep the ones I mentioned in the comment above

ahg-g · 2025-02-13T20:47:52Z

site-src/guides/dynamic-lora.md

+            ensureExist:
+            models:
+            - id: chatbot-v2
+                source: gs://[TEAM-A-MODELS-BUCKET]/chatbot-v2


need to update those, lets only work with hugging face adapters, otherwise it will be hard to run through the guide

site-src/guides/dynamic-lora.md

ahg-g · 2025-02-13T21:03:00Z

pkg/manifests/vllm/deployment-with-syncer.yaml

+          - '{"name": "tweet-summary-3", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'
+          - '{"name": "tweet-summary-4", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'
+          - '{"name": "sql-lora", "path": "yard1/llama-2-7b-sql-lora-test", "base_model_name": "llama-2"}'
+          - '{"name": "tweet-summary", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'


Lets only keep tweet-summary-0 and tweet-summary-1, and in the guide we show how we add tweet-summary-2 by modifying the inferenceModel they already created.

ahg-g · 2025-02-13T21:04:55Z

pkg/manifests/vllm/deployment-with-syncer.yaml

+          - base-model: meta-llama/Llama-2-7b-hf
+            id: sql-lora-v1
+            source: yard1/llama-2-7b-sql-lora-test
+          - base-model: meta-llama/Llama-2-7b-hf
+            id: sql-lora-v3
+            source: yard1/llama-2-7b-sql-lora-test
+          - base-model: meta-llama/Llama-2-7b-hf
+            id: sql-lora-v4
+            source: yard1/llama-2-7b-sql-lora-test


need to align this with https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/manifests/inferencemodel.yaml, so only keep the ones I mentioned in the comment above

ahg-g · 2025-02-13T21:05:41Z

site-src/guides/dynamic-lora.md

+                - id: tweet-summary-v1
+                    source: tweet-summary-1=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_1
+                - id: tweet-summary-v2
+                    source: tweet-summary-2=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_2


needs to be tweet-summary-0, tweet-summary-1 (existing) and a new tweet-summary-2

also, to update the configmap, we can instruct the user to:

kubectl edit dynamic-lora-config

site-src/guides/dynamic-lora.md

Signed-off-by: Daneyon Hansen <[email protected]>

Signed-off-by: Kunjan <[email protected]>

Makefile

site-src/guides/dynamic-lora.md

tools/dynamic-lora-sidecar/Makefile

Signed-off-by: Kunjan <[email protected]>

Makefile

pkg/manifests/vllm/deployment-with-syncer.yaml

site-src/guides/dynamic-lora.md

ahg-g · 2025-02-14T21:15:10Z

site-src/guides/dynamic-lora.md

+```yaml
+        apiVersion: v1
+        kind: ConfigMap
+        metadata:
+        name: dynamic-lora-config
+        data:
+        configmap.yaml: |
+             vLLMLoRAConfig:
+                name: sql-loras-llama
+                port: 8000
+                ensureExist:
+                    models:
+                    - base-model: meta-llama/Llama-2-7b-hf
+                      id: tweet-summary-0
+                      source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm
+                    - base-model: meta-llama/Llama-2-7b-hf
+                      id: tweet-summary-1
+                      source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm
+                    - base-model: meta-llama/Llama-2-7b-hf
+                      id: tweet-summary-2
+                      source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm


need to close this with ```, I couldn't make a suggestion to fix that.

ahg-g · 2025-02-14T21:18:27Z

/lgtm
/approve

This is great, thanks!

k8s-ci-robot · 2025-02-14T21:18:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, coolkp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 11, 2025

k8s-ci-robot requested review from ahg-g and liu-cong February 11, 2025 02:21

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 11, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 11, 2025

ahg-g reviewed Feb 11, 2025

View reviewed changes

site-src/guides/index.md Outdated Show resolved Hide resolved

site-src/guides/index.md Outdated Show resolved Hide resolved

coolkp force-pushed the lora-syncer-docs branch from a1fd513 to 69d965a Compare February 12, 2025 03:19

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 12, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 12, 2025

coolkp force-pushed the lora-syncer-docs branch from 69d965a to e43f5e9 Compare February 12, 2025 16:03

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 12, 2025

ahg-g reviewed Feb 12, 2025

View reviewed changes

site-src/guides/dynamic-lora.md Outdated Show resolved Hide resolved

coolkp force-pushed the lora-syncer-docs branch 2 times, most recently from e16a3ee to 9efc096 Compare February 12, 2025 16:22

ahg-g reviewed Feb 12, 2025

View reviewed changes

coolkp force-pushed the lora-syncer-docs branch 2 times, most recently from 8435721 to 310c5a8 Compare February 13, 2025 19:37

ahg-g reviewed Feb 13, 2025

View reviewed changes

coolkp force-pushed the lora-syncer-docs branch from ff6f5d5 to 0115291 Compare February 13, 2025 23:08

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 13, 2025

danehans and others added 2 commits February 13, 2025 15:48

Adds image-load and kind-load Make targets (kubernetes-sigs#288)

78b9bfe

Signed-off-by: Daneyon Hansen <[email protected]>

Add makefile and cloudbuild file to build and push lora-syncer

9c367f9

Signed-off-by: Kunjan <[email protected]>

coolkp force-pushed the lora-syncer-docs branch from 8fdf8a4 to 9c367f9 Compare February 13, 2025 23:57

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 13, 2025