[Feature] specify model in config.yaml

[wayzeng]

CLOSES #14819

Run CLI:

• vllm serve --config config.yaml with model in yaml.
• vllm serve SOME_MODEL --config config.yaml: model_tag in CLI, takes precedence over model in config file.
• If missing model in CLI and config file, will raise error.

Tests:

Unit tests:

✗ pytest test_utils.py
/tmp-nvme/vllm/lib/python3.12/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
============================================================================ test session starts =============================================================================
platform linux -- Python 3.12.9, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/jovyan/vllm
configfile: pyproject.toml
plugins: rerunfailures-14.0, anyio-4.6.2.post1, asyncio-0.24.0, shard-0.1.2, forked-1.6.0, mock-3.14.0, buildkite-test-collector-0.1.9
asyncio: mode=Mode.STRICT, default_loop_scope=None
collected 37 items                                                                                                                                                           
Running 37 items in this shard

test_utils.py .....................................                                                                                                                    [100%]

============================================================================== warnings summary ==============================================================================
tests/test_utils.py::test_memory_profiling
  /home/jovyan/vllm/tests/utils.py:669: DeprecationWarning: This process (pid=29729) is multi-threaded, use of fork() may lead to deadlocks in the child.
    pid = os.fork()

tests/test_utils.py::test_bind_kv_cache
tests/test_utils.py::test_bind_kv_cache
tests/test_utils.py::test_bind_kv_cache
tests/test_utils.py::test_bind_kv_cache
  /tmp-nvme/vllm/lib/python3.12/site-packages/triton/runtime/autotuner.py:108: DeprecationWarning: warmup, rep, and use_cuda_graph parameters are deprecated. See https://github.com/triton-lang/triton/pull/4496 for details.
    warnings.warn(("warmup, rep, and use_cuda_graph parameters are deprecated. See "

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================================================== 37 passed, 5 warnings in 15.02s =======================================================================

Integration tests

1. model in config file
   Create a test-config.yaml

model: meta-llama/Llama-3.1-8B-Instruct
uvicorn-log-level: "info"

Run vllm serve --config test-config.yaml
Can start the server and run benchmark.

2. model_tag in CLI, takes precedence over model in config file.
   Create a test-config.yaml

model: fake-model
uvicorn-log-level: "info"

Run vllm serve meta-llama/Llama-3.1-8B-Instruct --config test-config.yaml
Can start the server and run benchmark.

3. Missing model_tag in CLI and model in config file.

(vllm) ➜  vllm git:(update-cli) ✗ vllm serve --config test-config.yaml 
INFO 03-15 11:17:58 [__init__.py:256] Automatically detected platform cuda.
Traceback (most recent call last):
  File "/tmp-nvme/vllm/bin/vllm", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/jovyan/vllm/vllm/entrypoints/cli/main.py", line 70, in main
    args = parser.parse_args()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/vllm/vllm/utils.py", line 1196, in parse_args
    args = self._pull_args_from_config(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/vllm/vllm/utils.py", line 1288, in _pull_args_from_config
    raise ValueError(
ValueError: No model specified! Please specify model either in command-line arguments or in config file.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[aarnphm]

we should make a fixture factory to get data from tests/data in tests/conftest.py?

eventually I think it is good to migrate most test config to that folder

[wayzeng]

Hi @aarnphm, I moved the test configs fixture to tests/conftest.py, also changed the original data folder to config. Please let me know if any other issues. Thanks!

[aarnphm]

i have one questions, otherwise it looks good to me.

[aarnphm]

does this means vllm serve <model_name> --config config.yaml is allowed?

imo it should be either vllm serve <opts[...]> or vllm serve --config /path/to/yaml

[wayzeng]

Good question. My thought is for backward compatibility since vllm serve <model_name> --config config.yaml is the current conversion. It also aligns with the existing vllm doc: "In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence."

[aarnphm]

I see, that makes sense then.

[hmellor]

The flipping of this condition from != to == has broken v1-tests on main

[DarkLight1337]

I have reverted the commit, sorry for merging this (V1 test was failing for other reasons on main so I missed this). Please open a new commit that addresses this failure.

[wayzeng]

Hi all, I have fixed the failure and added more tests for handling --model error in #15798. Please take a look and let me know if any more issues. Thank you!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @wayzeng.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[ywang96]

Thanks for this QoL improvement!