[Bug]: [RLHF] Weights update broken with V1 multiprocessing

[22quinn]

Your current environment

wget https://raw.githubusercontent.com/vllm-project/vllm/main/collect_env.py
# For security purposes, please feel free to check the contents of collect_env.py before running it.
python collect_env.py

🐛 Describe the bug

Currently we update weights using load_weights API like here

LLMEngine.model_executor.driver_worker.worker.model_runner.model.load_weights(weights=[(name, weight)])

With V1 + multiprocessing, model_executor no longer exists: ref

I saw #14185 mentioned this caveat. Based on @youkaichao 's discussion in slack, it seems we had an implementation supporting multiprocessing but was blocked on some hang issue. Wondering what's the latest gap to fix this issue?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[rbao2018]

I think this path in verl may help
https://github.com/volcengine/verl/pull/923

[22quinn]

I think this path in verl may help https://github.com/volcengine/verl/pull/923

@rbao2018 Thanks for the pointer but in that PR, it still uses model = self.inference_engine.llm_engine.model_executor.driver_worker.worker.model_runner.model so I think will still fail if we turn on multiprocessing due to missing model_executor

[22quinn]

actually instead of directly calling LLMEngine.model_executor.driver_worker.worker.model_runner.model.load_weights, using collective_rpc("update_weight"...) might work, as implemented in #15444

model_executor.collective_rpc will be called here:

vllm/vllm/v1/engine/core.py

Line 290 in 3c0ff91

return self.model_executor.collective_rpc(method, timeout, args,