[Model] Support VLMs with transformers backend

[zucchini-nlp]

This PR adds support for multimodal models in Transformers backend. As a start I tested with vanilla LLaVA using demo scripts from the documentation. The generated outputs matched with VLLM outputs.

For this branch to work, we first need a few changes from transformers starting from huggingface/transformers#36367. Currently I want to ask for feedback, if this aligns with how VLLM sees things

cc @Isotr0py @ArthurZucker

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

Thanks for working on this! The main difficulty of supporting VLMs is not the model implementation itself, but rather the preprocessing code - vLLM V1 in particular requires precise tracking of the placeholder tokens. I see how generalizing return_token_type_ids to the multimodal context can help. We still have a couple of issues to tackle though:

• vLLM doesn't support non-consecutive multimodal placeholder feature tokens, which occurs when image tokens are arranged into a grid that is split by other padding tokens. We currently work around this in vLLM by considering those padding tokens as feature tokens as well, as demonstrated in our Fuyu example. To continue to support this workaround, we require information about the padding tokens when return_mm_token_type_ids=True.
• We need to also distinguish between tokens from different modalities when return_mm_token_type_ids=True. I suggest trying this with LLaVA-OneVision first.
• After applying HF processor, we split the output BatchFeature entries by their modality in order to cache them. To support this, we can maintain a mapping from the BatchFeature keys to their respective modalities as a class attribute in each HF processor.
• vLLM supports text+multimodal and token+multimodal inputs, but current HF processors only support the former. Is it feasible to also support token+multimodal inputs in HF Transformers? There are some cases where running tokenizer and HF processor on texts and multimodal data separately doesn't yield the same outputs as passing them together to HF processor, so we need to add special code in vLLM to handle this by overriding _apply_hf_processor_tokens_only.

cc @ywang96

[zucchini-nlp]

@DarkLight1337 thanks for review! Yeah, checking on more involved models is a good idea to verify all edge cases are covered, will do so. A few clarifications before that:

• For LLaVA-OneVision do we need to support inputs with video and image within one batch? If yes, that would complicate things a bit probably
• When doing token+multimodal inputs for the processor, do we expect the tokens to be already expanded or anyhow processed for multimodality? Or we assume that tokens are simply text+tokenizer output? I am not sure if it is a good idea to support it as part of call, but we can add a private method for VLLM to use. Also it would give us freedom to change API without caring for BC in the future, if we decide to handle tokens differently

[DarkLight1337]

For LLaVA-OneVision do we need to support inputs with video and image within one batch? If yes, that would complicate things a bit probably

Yes, we currently support mixed-modality (non-interleaved) inputs and plan to eventually support interleaved-modality inputs as well.

When doing token+multimodal inputs for the processor, do we expect the tokens to be already expanded or anyhow processed for multimodality? Or we assume that tokens are simply text+tokenizer output? I am not sure if it is a good idea to support it as part of call, but we can add a private method for VLLM to use. Also it would give us freedom to change API without caring for BC in the future, if we decide to handle tokens differently

We assume that the tokens have only gone through the tokenizer. So, placeholder tokens still have to be inserted into the input tokens. It's fine if we leave this unsolved for now - we can fall back to detokenizing the tokens back into text before passing them through HF processor.

[ywang96]

Thanks for the PR @zucchini-nlp! I'm a bit occupied at the moment but will take a first pass later tonight.

[ArthurZucker]

Have no say in this but that I am excited! 🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zucchini-nlp.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zucchini-nlp.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zucchini-nlp.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork