[Frontend] Add backend-specific options for guided decoding

[joerunde]

This PR extends the guided decoding backend name to support a list of backend-specific options. These are specified in a comma separated list after a colon. This allows us to easily add support for backend-specific features, like xgrammar:json-any-whitespace.

Specifically this PR also implements the no-fallback option. When set vLLM will return a very nicely formatted 400 describing how the guided decoding backend doesn't support what they asked for, instead of silently switching to a different backend. This is useful for users who want to only support a single guided decoding backend.

A motivation for this was that a product that only intended to support the xgrammar backend had a user find a query that would fall back to outlines, and then cause outlines to hang indefinitely. (See dottxt-ai/outlines-core#180)

Works around #12005 😉

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[markmc]

Have we an established pattern on what should be config vs env variables? Why wouldn't this be in DecodingConfig? Maybe we could encode "don't fallback" in something like --guided-decoding-backend=outlines:nofallback if we were worried about a proliferation of CLI arguments.

[mgoin]

I'm okay with either way, but I do think Mark's suggestion would be nicer. I like calling more attention to the --guided-decoding-backend argument if users want to be explicit about their backend

[russellb]

i like the cli arg approach ... before seeing this comment I was thinking about another "backend" like xgrammar-only or something like that. xgrammar:nofallback leaves it open to a bit more flexibility to specify additional options if necessary later, like xgrammar:nofallback,json-any-whitespace to support the case covered in #12744

[russellb]

Have we an established pattern on what should be config vs env variables?

Yeah ... I keep thinking about this. It's going to be a big project, but we're due for significant cleanup here. I'd really like a system that supports both config files and command line args (and less env vars unless it's just an alternative for setting the same set of options).

... but I have no idea when that's going to feel like the most important thing to work on!

[wallashss]

Have we an established pattern on what should be config vs env variables?

Yeah ... I keep thinking about this.

Me too. I like to think that the environment variables change the 'behavior' of the system like using a deprecated | experimental | workaround feature. While config are the others 'common features' of the system that's up to the users to set or tune to their environment.

However there also something that makes sense to this discussion. When we set in the config we have a chance to log the system setup, like the log Initializing a V0 LLM engine (v%s) with config: [...] Sometimes it is tricky to get the exact setup of the system when we got a crash and the only thing that we get it is a stack trace (which may be truncated as well 😄) .

Probably we should prefer using args before envs , but when makes sense to use envs, we probably could log to the users (at least once) that vLLM has this feature on and the implications of that in the system.

Either way: I like the idea of --guided-decoding-backend=outlines:nofallback for this PR. And I'm pretty sure that it would be logged in the system initialization, which is nice for debugging purpose.

[joerunde]

Thanks for the discussion everybody 👍

I stuck this in the environment to avoid the proliferation of cli args, but I love the suggestion of encoding the fallback behavior in the name of the backend. Best of both worlds!

I'll update the implementation

[joerunde]

Code is updated if y'all wanna take a second look 🙏

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @joerunde.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[russellb]

It will soon! :-). #13228

This is totally fine, though. I can tweak this once we're able to turn it on.

[russellb]

looks great to me! Thank you!

It would be good to update the text that shows up under vllm serve --help as well.

...
  --guided-decoding-backend {outlines,lm-format-enforcer,xgrammar}
                        Which engine will be used for guided decoding (JSON schema / regex etc) by default. Currently
                        support https://github.com/outlines-dev/outlines, https://github.com/mlc-ai/xgrammar, and
                        https://github.com/noamgat/lm-format-enforcer. Can be overridden per request via
                        guided_decoding_backend parameter.
...

[joerunde]

@russellb good catch, the CLI parser needed a bit of relaxing too.

@wallashss This now boots and logs the backend and option(s):

$ vllm serve Qwen/Qwen2.5-3B-Instruct --guided-decoding-backend xgrammar:no-fallback
... 
INFO 02-20 16:27:30 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.2.dev56+g49ad2b1b7.d20250211) with config: model='Qwen/Qwen2.5-3B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-3B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar:no-fallback'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen/Qwen2.5-3B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 

[mgoin]

Nice work, LGTM!