Add llama4

[ArthurZucker]

What does this PR do?

[yeqcharlotte]

Thanks!! 🔥🔥🔥🔥🔥

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[kadirnar]

@ArthurZucker

I ran the https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct model on a 1xA100 device. However, I'm getting this error.

  File "/ephemeral/.venv/lib/python3.10/site-packages/torch/nn/functional.py", line 5209, in pad
    return torch._C._nn.pad(input, pad, mode, value)
TypeError: pad(): argument 'pad' failed to unpack the object at pos 2 with error "type must be tuple of ints,but got NoneType"

Code:

from transformers import AutoProcessor, Llama4ForConditionalGeneration
import torch

model_id = "meta-llama/Llama-4-Maverick-17B-128E-Instruct"

processor = AutoProcessor.from_pretrained(model_id)
model = Llama4ForConditionalGeneration.from_pretrained(
    model_id,
    attn_implementation="flex_attention",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

url1 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
url2 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/cat_style_layout.png"
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": url1},
            {"type": "image", "url": url2},
            {"type": "text", "text": "Can you describe how these two images are similar, and how they differ?"},
        ]
    },
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
)

response = processor.batch_decode(outputs[:, inputs["input_ids"].shape[-1]:])[0]
print(response)
print(outputs[0])

env:

• transformers version: 4.51.0
• Platform: Linux-6.11.0-13-generic-x86_64-with-glibc2.40
• Python version: 3.10.16
• Huggingface_hub version: 0.30.1
• Safetensors version: 0.5.3
• Accelerate version: 1.6.0
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (GPU?): 2.6.0+cu124 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:
• Using GPU in script?:
• GPU type: NVIDIA A100 80GB PCIe

[ArthurZucker]

Having a look asap!

[nivibilla]

@ArthurZucker I assume this was a mistake to leave in? 😅

transformers/src/transformers/models/llama4/convert_llama4_weights_to_hf.py

Line 691 in d1b9236

default="/fsx/arthur/Llama-4-17B-Omni-Instruct-Original",

[ArthurZucker]

Oh yeah you are using dynamic cache we will disable it

[ArthurZucker]

Static cache should be used

[ddh0]

Is llama4 fully supported by Tranformers at this time? It would be nice to get some clarification on this since nearly everyone seems to be getting such bad results with Scout and Maverick

[ArthurZucker]

It is it is, we did not see that bad results but we are investigating!

[ArthurZucker]

😭

[radoslav-dimitrov-indeavr]

This PR causes Transformers to error out when a model is using Tensorflow and the environment does not provide torch in any way

transformers/src/transformers/pipelines/base.py:

        if torch.distributed.is_initialized():

Source

[YenFuLin]

Hi @ArthurZucker, why this modification is for llama4?

[ArthurZucker]

this was mostly because llama4 is too big to run without distributed, sorry that it broke stuff!