Skip to content

Commit e52baf6

Browse files
houseroadrasmith
authored andcommitted
[Bugfix] Check chain_speculative_sampling before calling it (vllm-project#11673)
Signed-off-by: Lu Fang <[email protected]>
1 parent 7cb0f4f commit e52baf6

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

vllm/model_executor/layers/rejection_sampler.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,7 @@ def forward(
118118

119119
# If use Flashinfer chain_speculative_sampling kernel
120120
# for rejection sampling
121-
if self.use_flashinfer:
121+
if self.use_flashinfer and chain_speculative_sampling is not None:
122122
batch_size, k, _ = draft_probs.shape
123123
uniform_samples = self._create_uniform_samples(
124124
seeded_seqs, batch_size, k, draft_probs.device)

0 commit comments

Comments
 (0)