[Kernel] GGUF MoeVec kernel #16780
Merged
[Kernel] GGUF MoeVec kernel #16780
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: SzymonOzog <[email protected]>
Signed-off-by: SzymonOzog <[email protected]>
Signed-off-by: SzymonOzog <[email protected]>
Signed-off-by: SzymonOzog <[email protected]>
SzymonOzog
requested review from
tlrmchlsmth,
WoosukKwon,
mgoin and
robertgshaw2-redhat
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
Isotr0py
approved these changes
Apr 20, 2025
Comment on lines
+381
to
+384
| torch::Tensor ggml_moe_a8_vec(torch::Tensor X, // input | ||
| torch::Tensor W, // expert weights | ||
| torch::Tensor topk_ids, int64_t top_k, | ||
| int64_t type, int64_t row, int64_t tokens) { |
There was a problem hiding this comment.
Can we also update the GGUF kernel tests to cover I-Quants with MoeVec kernel?
There was a problem hiding this comment.
Good idea. Updated with I-Quants
when I set max_model_len to 8192, The service will crash when it start error log |
It's work well，Below are the benchmark results on 8L40s45GB Benchmark result: @SzymonOzog Very fast, thanks for your work |
Signed-off-by: SzymonOzog <[email protected]>
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: SzymonOzog <[email protected]>
|
Can you merge from main to fix docker build? |
Signed-off-by: SzymonOzog <[email protected]>
|
@DarkLight1337 Merged main |
|
PTAL at the failing installation test. It seems related to this PR |
|
@DarkLight1337 So the test seems to use precompiled nightly wheen where the kernel from this PR is not yet present that's why |
|
@khluu @mgoin @LucasWilkinson any ideas? |
|
@SzymonOzog Can you merge from main to see if the python-only-installation test still fails? |
Signed-off-by: SzymonOzog <[email protected]>
|
Updated to main |
Signed-off-by: Isotr0py <[email protected]>
RichardoMrMu
pushed a commit
to RichardoMrMu/vllm
that referenced
this pull request
May 12, 2025
Signed-off-by: SzymonOzog <[email protected]> Signed-off-by: SzymonOzog <[email protected]> Signed-off-by: Isotr0py <[email protected]> Co-authored-by: Isotr0py <[email protected]> Signed-off-by: Mu Huai <[email protected]>
mawong-amd
pushed a commit
to ROCm/vllm
that referenced
this pull request
May 14, 2025
Signed-off-by: SzymonOzog <[email protected]> Signed-off-by: SzymonOzog <[email protected]> Signed-off-by: Isotr0py <[email protected]> Co-authored-by: Isotr0py <[email protected]>
zzzyq
pushed a commit
to zzzyq/vllm
that referenced
this pull request
May 24, 2025
Signed-off-by: SzymonOzog <[email protected]> Signed-off-by: SzymonOzog <[email protected]> Signed-off-by: Isotr0py <[email protected]> Co-authored-by: Isotr0py <[email protected]> Signed-off-by: Yuqi Zhang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When we don't have a high expert utilisation this kernel will work much faster than matmul style moe kernel. Also adds better support for I quants