【性能分析】这几天研究了一下nsys profile，尝试分析目前的generate时间瓶颈，分享下结论，不保证正确，希望各位讨论

[tarjintor]

由于kt用了cuda graph加速，简单的profiler似乎看不到graph内部的步骤时长，所以我用了nsys profile
如果把每一层都统计速度，数据量太大了，我就简单的开头，中间，结束各取了几层
需要的步骤包括

1. 改源码，我用的是0.2.0版本，引入import torch.cuda.nvtx as nvtx，然后在experts.py中KDeepseekV3MoE.forward中插入代码
   分别检查gate

        target_layers = [3, 22, 50]            
        if self.layer_idx_nvtx in target_layers:
            nvtx.range_push(f"moe_gate_{self.layer_idx_nvtx}")
        topk_idx, topk_weight = self.gate(hidden_states)
        if self.layer_idx_nvtx in target_layers:
            nvtx.range_pop()

检查路由专家，注意这里只是异步提交一个cpu任务，而不是任务完成了

            if self.layer_idx_nvtx in target_layers:
                nvtx.range_push(f"cpuinfer_{self.layer_idx_nvtx}")
            self.experts.generate_experts.submit_for_one_decode(hidden_states[0], topk_idx[0], topk_weight[0])
            if self.layer_idx_nvtx in target_layers:
                nvtx.range_pop()

检查共享专家

            if self.config.n_shared_experts is not None:
                if self.layer_idx_nvtx in target_layers:
                    nvtx.range_push(f"shared_experts_{self.layer_idx_nvtx}")
                y_ = self.shared_experts(identity).squeeze(0)
                if self.layer_idx_nvtx in target_layers:
                    nvtx.range_pop()

检查从cpu取回路由专家的结果

            if self.layer_idx_nvtx in target_layers:
                nvtx.range_push(f"self.experts.generate_experts.sync_for_one_decode {self.layer_idx_nvtx}")                                    
            y = self.experts.generate_experts.sync_for_one_decode().unsqueeze(0)
            if self.layer_idx_nvtx in target_layers:
                nvtx.range_pop()

然后作为对比，把attention的代码也监控起来，在modeling_deepseek_v3.py的DeepseekV3DecoderLayer的forward中添加

 target_layer = [2,3,4, 22,23, 50, 51]
        # Self Attention
        if self.layer_idx_nvtx in target_layer:
            nvtx.range_push(f"self_attention_{self.layer_idx_nvtx}")
        # if hasattr(self, "logger"):
        #     self.logger.info("------------ nvtx.range_push self_attn")
        hidden_states, self_attn_weights, present_key_value = self.self_attn(
            hidden_states=hidden_states,
            attention_mask=attention_mask,
            position_ids=position_ids,
            past_key_value=past_key_value,
            output_attentions=output_attentions,
            use_cache=use_cache,
            cache_position=cache_position,
            **kwargs,
        )
        if self.layer_idx_nvtx in target_layer:
            nvtx.range_pop()

然后执行命令
nsys profile --trace=cuda,nvtx,osrt,cudnn --cuda-graph-trace=node -o report python model_exam.py

这里model_exam.py可以改一下local_chat.py，不要轮询input，直接content是‘你好’来跑一轮对话再break即可
就可以得到一个分析结果，文件有点大，我就简单发了个截图
我的配置是ubuntu22.04,cuda12.2,python3.11,3960x用的20core，256g内存，2.51bit量化版本，4090显卡
这个图大概看上去每一层attention和moe的用时比例在2：3，所以我理解简单的说加快moe速度或者gpu速度可能都容易有边界效应，对我而言，先加快moe速度边界收益应该更高，但cpuinfer速度在整个计算中占比变低之后，也许能优化gpu算法会更好，当然，普通人4090已经够好了
抛砖引玉，求讨论

[Atream]

这个cutlass为啥这么大，你用的是windows或者ampere之前的GPU吗

[tarjintor]

这个cutlass为啥这么大，你用的是windows或者ampere之前的GPU吗

ubuntu22.04+4090

[tarjintor]

in case you are interesting in this @ubergarm

[Flying-Cloud]

Impressive！比较好奇您profile的这一层里Attention计算的token数是多少，我理解如果content输入只是“你好”而Attention占比如此之高，后续generate到100-1k token时的瓶颈应该会变成GPU

[tarjintor]

Impressive！比较好奇您profile的这一层里Attention计算的token数是多少，我理解如果content输入只是“你好”而Attention占比如此之高，后续generate到100-1k token时的瓶颈应该会变成GPU

我看了下后续变化也不大，我取的profile是generate阶段某一个token生成过程中，它走的61层中的第3,22,50层，从0开始计数
我的推理速度如下

2025-03-06 15:11:33,499 - mmlu_dev - INFO  - prefill_and_generate - prompt eval count:    6 token(s)
2025-03-06 15:11:33,499 - mmlu_dev - INFO  - prefill_and_generate - prompt eval duration: 1.283428430557251s
2025-03-06 15:11:33,499 - mmlu_dev - INFO  - prefill_and_generate - prompt eval rate:     4.674978251334875 tokens/s
2025-03-06 15:11:33,500 - mmlu_dev - INFO  - prefill_and_generate - eval count:           311 token(s)
2025-03-06 15:11:33,500 - mmlu_dev - INFO  - prefill_and_generate - eval duration:        47.40807294845581s
2025-03-06 15:11:33,500 - mmlu_dev - INFO  - prefill_and_generate - eval rate:            6.560064154856772 tokens/s

下图是generate接近尾声时候的截图，差别也不大看着

[ubergarm]

@tarjintor

惊人的实验结果！感谢您使用nsys profile工具进行的深入分析！

对于许多系统配置而言，优先优化运行在CPU上的MoE（混合专家模型）确实具有实际意义。这一点非常合理。

我注意到，在llama.cpp项目中，开发者正在尝试实现类似ktransformers的**数据并行（Data Parallel）**方案，以加速双路CPU或双NUMA节点配置。不过目前尚未看到针对MoE的专门优化进展。

在vLLM团队最近的办公时间视频（37:50处）中，他们重点讨论了数据并行与专家并行（Expert Parallelism）的协同优化设计，具体涉及此专家并行PR。随后在46:00处展示了昨日刚合并的PR，这可能是为MoE优化铺路。

有趣的是，vLLM目前测试使用的是IBM的小型MoE模型ibm-research/PowerMoE-3b。相较于R1 671B这类巨型模型，该模型的轻量化特性显然更便于快速验证和性能剖析。

最后，我不确定DeepSeek的EP如何应用于CPU/GPU系统，鉴于其专为通过纯依赖RDMA的专用低延迟内核实现多GPU推理而设计。我需要进一步深入研究。

这真是一个令人振奋的技术变革时代！

---

tarjintor

Wow, very interesting results. Thanks for the deep dive lesson using nsys profile!

That makes sense that MoE running on CPU might be optimization priority for many system configurations.

I've noticed that there is some interest in llama.cpp project to implement Data Parallel similar to ktransformers to speed up dual socket or two NUMA node configuration. I have not noticed any movement on specific MoE optimizations there yet.

I've noticed that the vLLM team is looking specifically at combination Data Parallel and Expert Parallelism design optimizations in their recent office hours YT video at time 37:50 specific to this expert parallel PR and again at time 46:00 with a PR just merged yesterday.

It seems they are possibly testing on a very small MoE ibm-research/PowerMoE-3b which is likely easier and faster to test and profile than R1 671B.

Finally, I'm not sure how DeepSeek's EP applies in a CPU/GPU system given it is specialized for multi-GPU inferencing using specialized low-latency kernels that rely purely on RDMA. I must study some more.

What an interesting time to be alive.

[yhfgyyf]

很棒的测试，需要对比下纯GPU推理下的各部分占比情况，我用6卡4090 48g，llama.cpp推理2.22bit 模型可以达到30t/s。ktranformers推理中gpu占用并不高，所以纯GPU推理可能expert计算的耗时会短很多

[ubergarm]

@yhfgyyf

很棒的测试，需要对比下纯GPU推理下的各部分占比情况，我用6卡4090 48g，llama.cpp推理2.22bit 模型可以达到30t/s。ktranformers推理中gpu占用并不高，所以纯GPU推理可能expert计算的耗时会短很多

老兄你运气真好，我们美国这边连48GB和96GB的4090都难搞到，更别说5090了——不过讲真，谁想要个只有36GB显存的卡啊？哈哈......

你试过针对自己的配置创建自定义卸载配置吗？需要花点功夫理解正则表达式并选择合适的模型层。可以参考现有的4卡/8卡GPU yaml文件示例，感兴趣的话不妨试试看。

$ cd ktransformers/optimize/optimize_rules/
$ cp DeepSeek-V3-Chat-multi-gpu-8.yaml DeepSeek-V3-Chat-multi-gpu-6.yaml
$ vim DeepSeek-V3-Chat-multi-gpu-6.yaml
$ ... --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-6.yaml

---

yhfgyyf

This is excellent testing. A comparison of the performance/resource allocation across components in pure GPU inference is needed. With 6x RTX 4090 (48GB each), I achieved 30 tokens/sec (t/s) using llama.cpp to infer a 2-bit quantized model. In ktranformers-based inference, GPU utilization remains low, suggesting that pure GPU inference might drastically reduce latency for expert computation steps.

Lucky guy, I wish we had the 48GB and 96GB 4090's more available here in USA. We can't even buy a 5090, but honestly who would want it anyway with only 36GB VRAM? hahaha...

Have you created a custom offload profile specific to your configuration? It will take some work for you to understand the regex and select appropriate layers. Look at the existing 4x and 8x GPU yaml files for examples and give it a try if you want.

$ cd ktransformers/optimize/optimize_rules/
$ cp DeepSeek-V3-Chat-multi-gpu-8.yaml DeepSeek-V3-Chat-multi-gpu-6.yaml
$ vim DeepSeek-V3-Chat-multi-gpu-6.yaml
$ ... --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-6.yaml