tool calling eval #256
Open
tool calling eval #256
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…xpected to trigger its tool call.
…ests LlamaStack models against various tools and generates detailed performance plots, including per-tool analysis.
…uation. These tools enable structured metric tracking to CSV and dynamic visualization of performance through generated plots. Added newline to end of queries.json
Ibrahim-Haroon
force-pushed
the
ibrahim/tool-calling-eval
branch
from
3785414 to
564c494
Compare
MichaelClifford
requested changes
May 30, 2025
tests/tool_calling/src/tools.py
Outdated
Comment on lines
10
to
11
| :description: Adds two numbers. | ||
| :use_case: Use when the user wants to find the sum, total, or combined value of two numbers. |
There was a problem hiding this comment.
I'm not sure if you need the :description:, :use_case: or :returns: ( at least according do the client tool decorator docstring). Have you found this to be superior to a plain docstring function description?
There was a problem hiding this comment.
At first I added :description: and :use_case: tags because I thought it would need it, didn't look at the docs. But ya, looking at client_tool.py it even dumps information about the return information on purpose, so that won't be needed. From limited experimentation it seems :description: and :use_case: slightly increase tool trivial accuracy.
experiment.pdf
Ibrahim-Haroon
force-pushed
the
ibrahim/tool-calling-eval
branch
3 times, most recently
from
ad5f2bd to
3f3142a
Compare
Ibrahim-Haroon
force-pushed
the
ibrahim/tool-calling-eval
branch
from
3f3142a to
e9dd497
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Used Gemini to create 40 simple functions that will be passed as tools. Created a
queries.jsonwhich has 15 queries per function which will be used as an "eval set" to determine accuracy of tool selection. Similar code totests/eval_tests/tests.pybut after runningeval.pyit shows successful calls per tool and not overall.How Has This Been Tested?
Merge criteria: