Skip to content

[Bug]: Qwen2.5-VL-72B Inference #16997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
1 task done
uyzhang opened this issue Apr 22, 2025 · 2 comments · Fixed by #16998
Closed
1 task done

[Bug]: Qwen2.5-VL-72B Inference #16997

uyzhang opened this issue Apr 22, 2025 · 2 comments · Fixed by #16998
Labels
bug Something isn't working

Comments

@uyzhang
Copy link

uyzhang commented Apr 22, 2025 •
edited
Loading

Your current environment

The output of `python collect_env.py` 
INFO 04-23 01:15:05 [__init__.py:239] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: TencentOS Server 3.2 (Final) (x86_64)
GCC version: (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
Clang version: 16.0.6 (Red Hat 16.0.6-2.module+el8.8.0+557+454507bd)
CMake version: version 3.28.0
Libc version: glibc-2.28

Python version: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.241-1-tlinux4-0017.7-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 12.8.61
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA H20
GPU 1: NVIDIA H20
GPU 2: NVIDIA H20
GPU 3: NVIDIA H20
GPU 4: NVIDIA H20
GPU 5: NVIDIA H20
GPU 6: NVIDIA H20
GPU 7: NVIDIA H20

Nvidia driver version: 535.161.08
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.9.7.1
/usr/lib64/libcudnn_adv.so.9.7.1
/usr/lib64/libcudnn_cnn.so.9.7.1
/usr/lib64/libcudnn_engines_precompiled.so.9.7.1
/usr/lib64/libcudnn_engines_runtime_compiled.so.9.7.1
/usr/lib64/libcudnn_graph.so.9.7.1
/usr/lib64/libcudnn_heuristic.so.9.7.1
/usr/lib64/libcudnn_ops.so.9.7.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              384
On-line CPU(s) list: 0-383
Thread(s) per core:  2
Core(s) per socket:  96
Socket(s):           2
NUMA node(s):        2
Vendor ID:           AuthenticAMD
BIOS Vendor ID:      Advanced Micro Devices, Inc.
CPU family:          25
Model:               17
Model name:          AMD EPYC 9K84 96-Core Processor
BIOS Model name:     AMD EPYC 9K84 96-Core Processor                
Stepping:            1
CPU MHz:             3700.055
CPU max MHz:         2600.0000
CPU min MHz:         1500.0000
BogoMIPS:            5200.20
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            32768K
NUMA node0 CPU(s):   0-95,192-287
NUMA node1 CPU(s):   96-191,288-383
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d

Versions of relevant libraries:
[pip3] numpy==2.2.5
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.4.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.51.3
[pip3] triton==3.2.0
[conda] numpy                     2.2.5                    pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-cusparselt-cu12    0.6.2                    pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] pyzmq                     26.4.0                   pypi_0    pypi
[conda] torch                     2.6.0                    pypi_0    pypi
[conda] torchaudio                2.6.0                    pypi_0    pypi
[conda] torchvision               0.21.0                   pypi_0    pypi
[conda] transformers              4.51.3                   pypi_0    pypi
[conda] triton                    3.2.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.5.dev149+gf34410715 (git sha: f34410715)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
	�[4mGPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	NIC0	NIC1	NIC2	NIC3	NIC4	NIC5	NIC6	NIC7	NIC8	NIC9	NIC10	NIC11	NIC12	NIC13	NIC14	NIC15	NIC16	NIC17	NIC18	NIC19	NIC20	NIC21	NIC22	NIC23	NIC24	NIC25	CPU Affinity	NUMA Affinity	GPU NUMA ID�[0m
GPU0	 X 	NV18	NV18	NV18	NV18	NV18	NV18	NV18	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	NODE	NODE	NODE	SYS	SYS	SYS	SYS	0-95,192-287	0		N/A
GPU1	NV18	 X 	NV18	NV18	NV18	NV18	NV18	NV18	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	PHB	PIX	SYS	SYS	SYS	SYS	0-95,192-287	0		N/A
GPU2	NV18	NV18	 X 	NV18	NV18	NV18	NV18	NV18	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	PIX	PHB	SYS	SYS	SYS	SYS	0-95,192-287	0		N/A
GPU3	NV18	NV18	NV18	 X 	NV18	NV18	NV18	NV18	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	NODE	PIX	NODE	NODE	SYS	SYS	SYS	SYS	0-95,192-287	0		N/A
GPU4	NV18	NV18	NV18	NV18	 X 	NV18	NV18	NV18	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	NODE	NODE	PIX	NODE	96-191,288-383	1		N/A
GPU5	NV18	NV18	NV18	NV18	NV18	 X 	NV18	NV18	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	NODE	PIX	NODE	NODE	96-191,288-383	1		N/A
GPU6	NV18	NV18	NV18	NV18	NV18	NV18	 X 	NV18	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	PHB	NODE	NODE	PIX	96-191,288-383	1		N/A
GPU7	NV18	NV18	NV18	NV18	NV18	NV18	NV18	 X 	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	PIX	NODE	NODE	PHB	96-191,288-383	1		N/A
NIC0	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC1	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC2	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC3	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC4	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC5	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC6	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC7	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC8	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC9	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC10	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC11	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC12	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC13	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC14	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC15	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC16	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC17	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	PIX	 X 	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE				
NIC18	PIX	NODE	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	NODE	NODE	NODE	SYS	SYS	SYS	SYS				
NIC19	NODE	NODE	NODE	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	NODE	 X 	NODE	NODE	SYS	SYS	SYS	SYS				
NIC20	NODE	PHB	PIX	NODE	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	 X 	PHB	SYS	SYS	SYS	SYS				
NIC21	NODE	PIX	PHB	NODE	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	PHB	 X 	SYS	SYS	SYS	SYS				
NIC22	SYS	SYS	SYS	SYS	NODE	NODE	PHB	PIX	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	 X 	NODE	NODE	PHB				
NIC23	SYS	SYS	SYS	SYS	NODE	PIX	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	NODE	 X 	NODE	NODE				
NIC24	SYS	SYS	SYS	SYS	PIX	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	NODE	NODE	 X 	NODE				
NIC25	SYS	SYS	SYS	SYS	NODE	NODE	PIX	PHB	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	PHB	NODE	NODE	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9
  NIC10: mlx5_10
  NIC11: mlx5_11
  NIC12: mlx5_12
  NIC13: mlx5_13
  NIC14: mlx5_14
  NIC15: mlx5_15
  NIC16: mlx5_16
  NIC17: mlx5_17
  NIC18: mlx5_bond_1
  NIC19: mlx5_bond_2
  NIC20: mlx5_bond_3
  NIC21: mlx5_bond_4
  NIC22: mlx5_bond_5
  NIC23: mlx5_bond_6
  NIC24: mlx5_bond_7
  NIC25: mlx5_bond_8

LD_LIBRARY_PATH=:/usr/local/nvshmem/lib:/opt/gdrcopy/lib/:/opt/gdrcopy/lib/:/opt/nvidia/nsight-systems/2024.6.2/host-linux-x64/:/opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst:/usr/local/cuda-12.8/lib64:
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

Which version of vLLM actually supports Qwen2.5-VL-72B inference, I've tried various versions since 0.8.2, including the latest commit tonight, and have had all sorts of problems with it, such as OUT OF MEMORY, etc. Also the offline version and the server version I've tried and have had all sorts of problems with both. So is there a normal usable version?

Some bug:
[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] WorkerProc hit an exception.
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] Traceback (most recent call last):
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 465, in worker_busy_loop
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] output = func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 263, in execute_model
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] output = self.model_runner.execute_model(scheduler_output)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1077, in execute_model
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] hidden_states = self.model(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1114, in forward
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] hidden_states = self.language_model.model(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 245, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] model_output = self.forward(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py", line 326, in forward
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] def forward(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return fn(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 822, in call_wrapped
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._wrapped_call(self, *args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 400, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] raise e
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 387, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc]
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "<eval_with_key>.162", line 574, in forward
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] submod_1 = self.submod_1(getitem, s0, getitem_1, getitem_2, getitem_3); getitem = getitem_1 = getitem_2 = submod_1 = None
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 822, in call_wrapped
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._wrapped_call(self, *args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 400, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] raise e
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 387, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc]
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "<eval_with_key>.2", line 5, in forward
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_1, 'language_model.model.layers.0.self_attn.attn'); query_2 = key_2 = value = output_1 = unified_attention_with_output = None
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/_ops.py", line 1123, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._op(*args, **(kwargs or {}))
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/attention/layer.py", line 415, in unified_attention_with_output
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] self.impl.forward(self,
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 578, in forward
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] cascade_attention(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 710, in cascade_attention
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] prefix_output, prefix_lse = flash_attn_varlen_func(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 253, in flash_attn_varlen_func
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] out, softmax_lse, _, _ = torch.ops._vllm_fa3_C.fwd(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/_ops.py", line 1123, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._op(*args, **(kwargs or {}))
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] RuntimeError: scheduler_metadata must have shape (metadata_size)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] Traceback (most recent call last):
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 465, in worker_busy_loop
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] output = func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 263, in execute_model
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] output = self.model_runner.execute_model(scheduler_output)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return func(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1077, in execute_model
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] hidden_states = self.model(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1114, in forward
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] hidden_states = self.language_model.model(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 245, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] model_output = self.forward(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py", line 326, in forward
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] def forward(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return fn(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 822, in call_wrapped
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._wrapped_call(self, *args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 400, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] raise e
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 387, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc]
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "<eval_with_key>.162", line 574, in forward
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] submod_1 = self.submod_1(getitem, s0, getitem_1, getitem_2, getitem_3); getitem = getitem_1 = getitem_2 = submod_1 = None
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 822, in call_wrapped
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._wrapped_call(self, *args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 400, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] raise e
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/fx/graph_module.py", line 387, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc]
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._call_impl(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return forward_call(*args, **kwargs)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "<eval_with_key>.2", line 5, in forward
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_1, 'language_model.model.layers.0.self_attn.attn'); query_2 = key_2 = value = output_1 = unified_attention_with_output = None
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/_ops.py", line 1123, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._op(*args, **(kwargs or {}))
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/attention/layer.py", line 415, in unified_attention_with_output
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] self.impl.forward(self,
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 578, in forward
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] cascade_attention(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 710, in cascade_attention
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] prefix_output, prefix_lse = flash_attn_varlen_func(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py", line 253, in flash_attn_varlen_func
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] out, softmax_lse, _, _ = torch.ops._vllm_fa3_C.fwd(
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] File "/jizhicfs/leoyizhang/anaconda3/envs/vllm_0.8.4_f344107/lib/python3.12/site-packages/torch/_ops.py", line 1123, in call
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] return self._op(*args, **(kwargs or {}))
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470] RuntimeError: scheduler_metadata must have shape (metadata_size)
�[1;36m(VllmWorker rank=3 pid=1863294)�[0;0m ERROR 04-23 01:03:33 [multiproc_executor.py:470]

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@uyzhang uyzhang added the bug Something isn't working label Apr 22, 2025
@uyzhang uyzhang changed the title [Bug]: Qwen2.5-VL-72B [Bug]: Qwen2.5-VL-72B Inference Apr 22, 2025
@DarkLight1337
Copy link
Member

Out of memory just means you don't have enough memory, it's not really a bug. You can try these steps to reduce memory usage: https://docs.vllm.ai/en/latest/serving/offline_inference.html#reducing-memory-usage

@DarkLight1337
Copy link
Member

The metadata issue should be fixed by #16998

@DarkLight1337 DarkLight1337 mentioned this issue Apr 23, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@uyzhang @DarkLight1337