Skip to content

LLMServerPool Implementation #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Nov 19, 2024
Merged

LLMServerPool Implementation #36

merged 1 commit into from
Nov 19, 2024

Conversation

kfswain
Copy link
Collaborator

@kfswain kfswain commented Oct 28, 2024 •
edited
Loading

Implementation to integrate LLMServerPool and Pod reconciliation into the I-GW EndpointPicker(ext-proc)

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 28, 2024
@k8s-ci-robot k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 28, 2024
@kfswain
Copy link
Collaborator Author

kfswain commented Oct 28, 2024

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 28, 2024
@kfswain
Copy link
Collaborator Author

kfswain commented Oct 28, 2024

/label tide/merge-method-squash

@k8s-ci-robot k8s-ci-robot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Oct 28, 2024
kfswain
kfswain commented Oct 29, 2024
View reviewed changes
@kfswain kfswain force-pushed the llmserverpool-implementation branch 2 times, most recently from 1a3ab93 to 790d3a2 Compare October 29, 2024 20:46
@kfswain
Copy link
Collaborator Author

kfswain commented Oct 29, 2024

/unhold

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 29, 2024
ahg-g
ahg-g reviewed Nov 4, 2024
View reviewed changes
Copy link
Contributor

@ahg-g ahg-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use controller-runtime instead of creating and managing workqueues and informerFactory etc. https://github.com/kubernetes-sigs/controller-runtime

See the example here, it is simpler: https://github.com/kubernetes-sigs/controller-runtime/blob/main/examples/crd/main.go

@kfswain kfswain force-pushed the llmserverpool-implementation branch 2 times, most recently from 7db0c98 to 741da3d Compare November 12, 2024 18:22
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 12, 2024
@kfswain kfswain force-pushed the llmserverpool-implementation branch 2 times, most recently from 42c2f72 to cc30945 Compare November 14, 2024 16:32
@kfswain kfswain force-pushed the llmserverpool-implementation branch from cc30945 to 583fd27 Compare November 17, 2024 16:30
@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 17, 2024
@kfswain kfswain force-pushed the llmserverpool-implementation branch from 583fd27 to e9617a9 Compare November 18, 2024 15:33
ahg-g
ahg-g reviewed Nov 18, 2024
View reviewed changes
Copy link
Contributor

@ahg-g ahg-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started to review this, but it seems this is still in development, right?

@ahg-g
Copy link
Contributor

ahg-g commented Nov 18, 2024

/hold

just so this doesn't get merged by mistake

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 18, 2024
@kfswain
Copy link
Collaborator Author

kfswain commented Nov 18, 2024

I started to review this, but it seems this is still in development, right?

Yes, I should have put a hold on it (ty for that). I'll take the hold off when it's complete.

@kfswain kfswain force-pushed the llmserverpool-implementation branch from e9617a9 to 133349f Compare November 18, 2024 18:45
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kfswain

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 18, 2024
ahg-g
ahg-g reviewed Nov 18, 2024
View reviewed changes
pkg/ext-proc/backend/llmserverpool_controller.go Outdated
}
klog.V(1).Info("reconciling LLMServerPool", req.NamespacedName)

serverPool := &v1alpha1.LLMServerPool{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aren't we supposed to create a Service here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized that it would be confusing to have the EPP reconcile and create an object on the LLMServerPool's behalf, in addition to whatever controller created the EPP. Instead, the implementation of the LLMServerPool controller should create the service and pass the service name in on startup. I think centralizing the object creation/control will make for a cleaner interface/experience

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this comment unresolved for posterity. But discussed further offline. And it sounds like we are in agreement here

@kfswain kfswain force-pushed the llmserverpool-implementation branch from 133349f to 72bcadc Compare November 18, 2024 20:44
@kfswain kfswain force-pushed the llmserverpool-implementation branch from 72bcadc to 37da3e5 Compare November 18, 2024 22:03
@kfswain
Copy link
Collaborator Author

kfswain commented Nov 18, 2024

/unhold

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 18, 2024
@ahg-g
Copy link
Contributor

ahg-g commented Nov 19, 2024

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 19, 2024
@k8s-ci-robot k8s-ci-robot merged commit f82445b into kubernetes-sigs:main Nov 19, 2024
2 checks passed
@kfswain kfswain deleted the llmserverpool-implementation branch November 19, 2024 19:58
@liu-cong liu-cong mentioned this pull request Nov 25, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ahg-g ahg-g ahg-g left review comments

@robscott robscott Awaiting requested review from robscott

Assignees

@ahg-g ahg-g

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kfswain @ahg-g @k8s-ci-robot