[Audio] Support Audio Datasets

[kylesayrs]

Purpose

• Support oneshot with audio datasets

Changes

• Extend apply_pad_mask_to_batch to handle cases where there are no input_ids and where there might be decoder_input_ids
• Extend TextGenerationDataset to detect if a dataset is already tokenized based on processor.model_input_names rather than only input_ids

Testing

• Ran test_processors.py to completion, which verifies that the model_input_names attribute is defined for most processors
• Ran whisper to completion in [Audio] Qwen Audio Example #1082

test_processors.py

import pytest
from transformers import AutoProcessor

@pytest.mark.parametrize(
    "model_id,expected",
    [
        ("meta-llama/Meta-Llama-3-8B-Instruct", ["input_ids", "attention_mask"]),
        ("mistralai/Mixtral-8x7B-Instruct-v0.1", ["input_ids", "attention_mask"]),
        (
            "Qwen/Qwen2-VL-2B-Instruct",
            [
                "input_ids",
                "attention_mask",
                "pixel_values",
                "image_grid_thw",
                "pixel_values_videos",
                "video_grid_thw",
            ],
        ),
        ("mgoin/pixtral-12b", ["input_ids", "attention_mask", "pixel_values"]),
        ("openai/whisper-large-v2", ["input_features"]),
        (
            "Qwen/Qwen2-Audio-7B-Instruct",
            ["input_ids", "attention_mask", "input_features", "feature_attention_mask"],
        ),
    ],
)
def test_processor_model_input_names(model_id, expected):
    """
    Tests the model_input_names attribute of common model processors
    """

    processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
    assert processor.model_input_names == expected

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.