Image Tool Use Result Causes Response Error in Anthropic API

[maxie]

I was trying to return an image to Anthropic as a tool use result, but the message sent to the Anthropic API looks like this, which results in a response error:

{
'role': 'user', 
'content': [{
        'type': 'tool_result', 
        'tool_use_id': 'toolu_0189y67pid1MQxkr6ytTH21h',
        'content': [ImageContent(type='image', data='iVBORw0K...........==', mimeType='image/png', annotations=None)], 
        'is_error': False,
  }]
}

I found that in augmented_llm_anthropic.py, after calling tools, the type of result.content added to the messages list does not seem to match the expected format:

result = await self.call_tool( 
    request=tool_call_request, tool_call_id=tool_use_id
)

messages.append(
    MessageParam(
        role="user",
        content=[
            ToolResultBlockParam(
                type="tool_result",
                tool_use_id=tool_use_id,
                content=result.content,   
                // result.content is list[TextContent | ImageContent | EmbeddedResource]
                // Expected : Union[str, Iterable[Content]], where Content is Union[TextBlockParam, ImageBlockParam]
                is_error=result.isError,
            )
        ],
    )
)

Is there a missing type conversion for result.content before adding it to the message? Or Could AnthropicMCPTypeConverter already be handling this conversion elsewhere without me knowing?

Thanks a lot!

[saqadri]

@maxie thanks for reporting. Can you please share repro steps? It's quite possible there's an issue in the type converters that needs to be fixed.

[maxie]

@saqadri I've written an example code to reproduce this issue. You can check it here: #100

By the way, I tried to write an example conversion method, and now the response from Anthropics is correct. However, I’m not sure if this is the right place to fix the issue.

@classmethod
def convert_call_tool_result_content_to_tool_result_block_param_content(
    cls, tool_result_content: list[TextContent | ImageContent | EmbeddedResource]
) -> Iterable[Union[TextBlockParam, ImageBlockParam]]:
    result_content = []

    for content in tool_result_content:
        if isinstance(content, TextContent):
            result_content.append(TextBlockParam(type=content.type, text=content.text))
        elif isinstance(content, ImageContent):
            result_content.append(ImageBlockParam(type="image", source=Base64ImageSourceParam(
                type="base64",
                media_type=content.mimeType,
                data=content.data
            )))
        elif isinstance(content, EmbeddedResource):
            if isinstance(content.resource, TextResourceContents):
                result_content.append(TextBlockParam(type="text", text=content.resource.text))
            else:  # BlobResourceContents
                result_content.append(TextBlockParam(
                    type="text", text=f"{content.resource.mimeType}:{content.resource.blob}"
                ))
        else:
            # Last effort to convert the content to a string
            result_content.append(TextBlockParam(type="text", text=str(content)))

    return result_content