[Bug]: Function calling with Qwen & Streaming ('NoneType' object has no attribute 'get')

[githebs]

Your current environment

The output of `python collect_env.py`

Your output of `python collect_env.py` here

Model Input Dumps

No response

🐛 Describe the bug

vLLM Version

v0.6.3.post1

Model

Qwen2.5-7B-Instruct

Docker command for vLLM

command: --host 0.0.0.0 --model /hf/Qwen-Qwen2.5-7B-Instruct --max-model-len 32768 --gpu_memory_utilization 0.9 --enable-auto-tool-choice --tool-call-parser hermes

Parsing from my own fastapi

async def stream_response(payload: dict, log: RequestLogger) -> AsyncGenerator[str, None]:
    """Handle streaming response from vLLM."""
    async with httpx.AsyncClient() as client:
        try:
            async with client.stream(
                'POST',
                VLLM_API_BASE,
                json=payload,
                headers={"Content-Type": "application/json"},
                timeout=30.0
            ) as response:
                if response.status_code != 200:
                    error_msg = f"vLLM API error: {response.status_code}"
                    log(error_msg, level='error')
                    yield f"data: {json.dumps({'error': error_msg})}\n\n"
                    return

                async for line in response.aiter_lines():
                    if not line or not line.startswith('data: '):
                        continue
                        
                    line = line.removeprefix('data: ')
                    if line.strip() == '[DONE]':
                        log("Stream completed")
                        yield 'data: [DONE]\n\n'
                        break
                    
                    try:
                        parsed = json.loads(line)
                        log("Streaming chunk", parsed)

                        # Handle tool calls in streaming response
                        if 'choices' in parsed and parsed['choices']:
                            choice = parsed['choices'][0]
                            if 'delta' in choice and 'tool_calls' in choice['delta']:
                                tool_call = choice['delta']['tool_calls'][0]
                                
                                if ('function' in tool_call and 
                                    'name' in tool_call['function'] and 
                                    'arguments' in tool_call['function']):
                                    
                                    func_name = tool_call['function']['name']
                                    args = json.loads(tool_call['function']['arguments'])
                                    
                                    if func_name == 'add_numbers':
                                        result = add_numbers(args['a'], args['b'])
                                        yield f'data: {json.dumps({"choices": [{"delta": {"content": str(result)}}]})}\n\n'
                                        continue

                        yield f'data: {line}\n\n'
                    except json.JSONDecodeError as e:
                        log(f"Failed to parse streaming response: {str(e)}", level='error')
                        continue

        except httpx.RequestError as e:
            error_msg = f"Streaming request failed: {str(e)}"
            log(error_msg, level='error')
            yield f"data: {json.dumps({'error': error_msg})}\n\n"
        
    log("Stream connection closed")

vLLM error

vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] Error trying to handle streaming tool call. vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] Traceback (most recent call last): vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py", line 226, in extract_tool_calls_streaming vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] function_name: Union[str, None] = current_tool_call.get("name") vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] ^^^^^^^^^^^^^^^^^^^^^ vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] AttributeError: 'NoneType' object has no attribute 'get'

Please note that everything works if

1. Streaming with no tools
2. Not streaming with tools

Any guidance ?
Thanks in advance everyone

PS: I have seen the posts from #9693 but my issue seems different since i actually use a "supported" model.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

cc @K-Mistele

[K-Mistele]

Thanks for the ping @DarkLight1337
@githebs can you share a request configuration that reproduces the issue consistently (temperature=0 is great for reproducibility, but no worries if you need a higher temp and it only happens sometimes) so that I can debug and take a look?

[K-Mistele]

Hi @githebs - we have had a discussion on this issue in #9693. Please see my comment here and let me know if this seems like a good path forward for you.

[K-Mistele]

Please check #9908 :)

[frei-x]

Stream output, if the function has no parameters, an error will be reported directly

[K-Mistele]

Stream output, if the function has no parameters, an error will be reported directly

Yeah, this is what I'm thinking too. #9908 (comment)

[githebs]

@frei-x @K-Mistele

thanks for the answer, sorry for the delay, I answered in the PR here #9908 (comment) but basically, yes, if the argument is blank, it doesn't work

[wangluyi]

In my case, when I use tool call and no argument is needed for the function, the code throws exception like this :

It appears that the bug is here:
chat_utils.py line 498:

def _postprocess_messages(messages: List[ConversationMessage]) -> None:
    # per the Transformers docs & maintainers, tool call arguments in
    # assistant-role messages with tool_calls need to be dicts not JSON str -
    # this is how tool-use chat templates will expect them moving forwards
    # so, for messages that have tool_calls, parse the string (which we get
    # from openAI format) to dict
    for message in messages:
        if (message["role"] == "assistant" and "tool_calls" in message
                and isinstance(message["tool_calls"], list)):

            for item in message["tool_calls"]:
                item["function"]["arguments"] = json.loads(
                    item["function"]["arguments"])

I modified the code like this (simply deal with blank argument) and it goes right:

def _postprocess_messages(messages: List[ConversationMessage]) -> None:
    # per the Transformers docs & maintainers, tool call arguments in
    # assistant-role messages with tool_calls need to be dicts not JSON str -
    # this is how tool-use chat templates will expect them moving forwards
    # so, for messages that have tool_calls, parse the string (which we get
    # from openAI format) to dict
    for message in messages:
        if (message["role"] == "assistant" and "tool_calls" in message
                and isinstance(message["tool_calls"], list)):

            for item in message["tool_calls"]:
                if item["function"] is not None and item["function"]["arguments"] is not None and len(item["function"]["arguments"]) > 0:
                    item["function"]["arguments"] = json.loads(item["function"]["arguments"])
                else:
                    item["function"]["arguments"] = {}

[K-Mistele]

Related #11522

[wey-gu]

Any heroes are working on this, please?🙏

Thanks!
cc @K-Mistele

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!