Empty args/inputs when turning streaming on and setting tool choice to any

[kwnath]

Description

When calling the Anthropic client with streaming=True and tool_choice={"type": "any"/"tool" } the output returns a tool call but with empty args.

This is problematic for a few other reasons beyond no args being returned. For example, quite a few packages rely on the anthropic-sdk, one of which is langchain-anthropic (ref).

Expected response

I would expect that the output includes the inputs/args required for the tool call when streaming.

Reproduction steps

I've added a notebook to highlight some things:
https://gist.github.com/kwnath/f42737c023767d5effdcca20cb5bd0a6

[RobertCraigie]

@kwnath thanks for the report and the detailed example notebook.

Can you share the python version you're using and the dependency versions you're using? e.g. with pip list.

I can't reproduce the issue you're seeing, with this script

from anthropic import Anthropic

client = Anthropic()

stream = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=2048,
    temperature=0.0,
    tools=[
        {
            "name": "get_weather",
            "description": "Get the current weather in a given location",
            "input_schema": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    }
                },
                "required": ["location"],
            },
        }
    ],
    tool_choice={"type": "any"},
    stream=True,
    messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
)
for event in stream:
    print(event)

I get back multiple events

RawMessageStartEvent(message=Message(id='msg_01177R2njzZanvg1kphN3Wvh', content=[], model='claude-3-5-sonnet-20240620', role='assistant', stop_reason=None, stop_sequence=None, type='message', usage=Usage(input_tokens=366, output_tokens=10)), type='message_start')
RawContentBlockStartEvent(content_block=ToolUseBlock(id='toolu_01SjuyZhYAert5cA4LNzNNKc', input={}, name='get_weather', type='tool_use'), index=0, type='content_block_start')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='{"', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='location": ', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='"San Fran', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='ci', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='sco, CA', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='"}', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockStopEvent(index=0, type='content_block_stop')
RawMessageDeltaEvent(delta=Delta(stop_reason='tool_use', stop_sequence=None), type='message_delta', usage=MessageDeltaUsage(output_tokens=41))
RawMessageStopEvent(type='message_stop')

[kwnath]

Ah completely forgot to add this! Apologies!

Python 3.11.7

Package                                      Version
-------------------------------------------- -----------
absl-py                                      1.4.0
aiohappyeyeballs                             2.4.0
aiohttp                                      3.10.5
aiosignal                                    1.2.0
alembic                                      1.13.1
altair                                       5.0.1
amazon-sqs-extended-client                   1.0.0
aniso8601                                    9.0.1
annotated-types                              0.6.0
ansiwrap                                     0.8.4
anthropic                                    0.34.2
anyio                                        3.6.2
appnope                                      0.1.3
arviz                                        0.15.1
asn1crypto                                   1.5.1
astroid                                      2.15.5
asttokens                                    2.0.5
astunparse                                   1.6.3
attrs                                        23.2.0
aws-sam-translator                           1.91.0
aws-xray-sdk                                 2.9.0
awswrangler                                  3.2.1
backcall                                     0.2.0
backoff                                      1.11.1
beautifulsoup4                               4.9.1
black                                        24.3.0
blinker                                      1.6.2
blis                                         0.7.9
boto3                                        1.34.162
botocore                                     1.34.162
Bottleneck                                   1.3.7
buildkite-test-collector                     0.1.8
cachetools                                   5.3.3
catalogue                                    2.0.8
certifi                                      2024.7.4
cffi                                         1.16.0
cfn-lint                                     1.10.3
charset-normalizer                           2.0.6
cityhash                                     0.4.7
click                                        8.1.6
cloudpathlib                                 0.16.0
cloudpickle                                  2.2.1
comm                                         0.1.3
confection                                   0.1.4
cons                                         0.4.6
contourpy                                    1.1.0
cryptography                                 42.0.5
cycler                                       0.10.0
cymem                                        2.0.7
dataclasses-json                             0.6.6
datadog                                      0.42.0
debugpy                                      1.6.7
decorator                                    5.1.1
defusedxml                                   0.7.1
Deprecated                                   1.2.13
dill                                         0.3.6
diskcache                                    5.4.0
distro                                       1.8.0
docker                                       5.0.3
ecdsa                                        0.19.0
elastic-transport                            8.4.0
elasticsearch                                8.8.2
en_core_web_lg                               3.7.1
entrypoints                                  0.3
etuples                                      0.3.9
executing                                    0.8.3
faiss-cpu                                    1.7.4
Faker                                        8.4.0
fastjsonschema                               2.18.0
fastprogress                                 1.0.3
fasttext                                     0.9.2
filelock                                     3.12.2
flake8                                       6.0.0
flake8-tidy-imports                          4.10.0
Flask                                        2.3.2
Flask-Cors                                   4.0.1
flask-konch                                  2.0.0
Flask-Migrate                                4.0.7
Flask-SQLAlchemy                             3.1.1
flatbuffers                                  23.5.26
fonttools                                    4.47.2
frozenlist                                   1.4.0
fsspec                                       2023.12.2
funcy                                        2.0
future                                       0.18.3
gast                                         0.4.0
gitdb                                        4.0.10
GitPython                                    3.1.41
google-pasta                                 0.2.0
googleapis-common-protos                     1.56.4
graphene                                     3.3
graphql-core                                 3.2.1
graphql-relay                                3.2.0
graphviz                                     0.13
grpcio                                       1.62.0
gunicorn                                     22.0.0
h11                                          0.14.0
h5netcdf                                     1.3.0
h5py                                         3.11.0
hanzidentifier                               1.2.0
httpcore                                     1.0.2
httpx                                        0.26.0
huggingface-hub                              0.20.2
idna                                         3.7
importlib-metadata                           6.8.0
iniconfig                                    1.1.1
ipdb                                         0.13.13
IProgress                                    0.4
ipykernel                                    6.25.0
ipython                                      8.12.2
ipywidgets                                   8.1.2
isort                                        5.12.0
itsdangerous                                 2.1.2
ja_core_news_sm                              3.7.0
jaraco.classes                               3.2.3
jedi                                         0.19.1
Jinja2                                       3.1.4
jiter                                        0.4.0
jmespath                                     0.10.0
joblib                                       1.3.2
jsondiff                                     2.0.0
jsonpatch                                    1.33
jsonpointer                                  2.3
jsonschema                                   4.23.0
jsonschema-path                              0.3.3
jsonschema-specifications                    2023.12.1
jupyter_client                               8.3.0
jupyter_core                                 5.3.1
jupyterlab_widgets                           3.0.10
jupytext                                     1.14.7
keras                                        3.3.3
keyring                                      24.2.0
kiwisolver                                   1.3.1
ko_core_news_sm                              3.7.0
konch                                        4.4.0
langchain                                    0.2.11
langchain-anthropic                          0.1.23
langchain-aws                                0.1.15
langchain-community                          0.2.9
langchain-core                               0.2.28
langchain-openai                             0.1.16
langchain-text-splitters                     0.2.0
langcodes                                    3.3.0
langsmith                                    0.1.77
lazy-object-proxy                            1.9.0
Levenshtein                                  0.21.0
libclang                                     16.0.0
llvmlite                                     0.41.1
logical-unification                          0.4.6
lxml                                         4.9.1
Mako                                         1.3.3
Markdown                                     3.4.3
markdown-it-py                               2.2.0
MarkupSafe                                   2.1.3
marshmallow                                  3.21.2
matplotlib                                   3.7.2
matplotlib-inline                            0.1.6
mccabe                                       0.7.0
mdit-py-plugins                              0.3.0
mdurl                                        0.1.1
miniKanren                                   1.0.3
mistune                                      3.0.1
ml-dtypes                                    0.3.2
mlflow                                       2.13.2
more-itertools                               8.7.0
moto                                         4.2.14
mpmath                                       1.3.0
multidict                                    6.0.2
multipledispatch                             1.0.0
multiprocess                                 0.70.14
murmurhash                                   1.0.9
mypy-extensions                              1.0.0
namex                                        0.0.8
nbclient                                     0.8.0
nbformat                                     5.9.1
nest-asyncio                                 1.5.6
networkx                                     2.6.3
nltk                                         3.6.7
numba                                        0.58.1
numexpr                                      2.8.6
numpy                                        1.23.5
openai                                       1.35.14
openapi-schema-validator                     0.6.2
openapi-spec-validator                       0.7.1
opentelemetry-api                            1.20.0
opentelemetry-exporter-otlp-proto-common     1.20.0
opentelemetry-exporter-otlp-proto-http       1.20.0
opentelemetry-instrumentation                0.41b0
opentelemetry-instrumentation-aiohttp-client 0.41b0
opentelemetry-instrumentation-botocore       0.41b0
opentelemetry-instrumentation-elasticsearch  0.41b0
opentelemetry-instrumentation-flask          0.41b0
opentelemetry-instrumentation-httpx          0.41b0
opentelemetry-instrumentation-requests       0.41b0
opentelemetry-instrumentation-sqlalchemy     0.41b0
opentelemetry-instrumentation-wsgi           0.41b0
opentelemetry-propagator-aws-xray            1.0.1
opentelemetry-proto                          1.20.0
opentelemetry-sdk                            1.20.0
opentelemetry-semantic-conventions           0.41b0
opentelemetry-util-http                      0.41b0
opt-einsum                                   3.3.0
optree                                       0.11.0
orjson                                       3.9.15
packaging                                    23.2
pandas                                       2.0.3
papermill                                    2.4.0
papermill_jupytext                           0.0.1
parso                                        0.8.3
pathable                                     0.4.3
pathlib2                                     2.3.7.post1
pathos                                       0.3.0
pathspec                                     0.11.1
patsy                                        0.5.6
pexpect                                      4.8.0
phonenumbers                                 8.13.31
pickleshare                                  0.7.5
pillow                                       10.3.0
pip                                          24.2
platformdirs                                 3.2.0
plotly                                       5.18.0
pluggy                                       0.13.1
pox                                          0.3.4
ppft                                         1.7.6.8
preshed                                      3.0.8
presidio-analyzer                            2.2.353
presidio-anonymizer                          2.2.353
prompt-toolkit                               3.0.39
protobuf                                     3.20.3
psutil                                       5.9.5
ptyprocess                                   0.7.0
pure-eval                                    0.2.2
py-partiql-parser                            0.5.0
pyarrow                                      14.0.1
pyasn1                                       0.6.0
pybind11                                     2.9.2
pycodestyle                                  2.10.0
pycparser                                    2.21
pycryptodome                                 3.20.0
pycurl                                       7.44.1
pydantic                                     2.6.0
pydantic_core                                2.16.1
pydeck                                       0.8.0
pyflakes                                     3.0.1
Pygments                                     2.16.1
PyJWT                                        2.8.0
pylint                                       2.17.4
pymc                                         5.10.4
Pympler                                      0.9
PyMySQL                                      1.1.1
pyOpenSSL                                    24.0.0
pyparsing                                    3.0.9
pytensor                                     2.18.6
pytest                                       7.2.0
pytest-asyncio                               0.21.0
pytest_httpx                                 0.29.0
pytest-split                                 0.9.0
python-dateutil                              2.8.2
python-dotenv                                0.21.1
python-jose                                  3.3.0
python-Levenshtein                           0.21.0
pytz                                         2021.1
PyYAML                                       6.0.1
pyzmq                                        25.1.0
querystring-parser                           1.2.4
rapidfuzz                                    3.6.2
referencing                                  0.35.1
regex                                        2023.12.25
requests                                     2.32.3
requests-file                                2.0.0
responses                                    0.25.3
rfc3339-validator                            0.1.4
rich                                         13.2.0
rpds-py                                      0.20.0
rsa                                          4.9
s3transfer                                   0.10.1
safetensors                                  0.4.1
sagemaker                                    2.218.1
sagemaker-mlflow                             0.1.0
schema                                       0.7.7
scikit-learn                                 1.3.1
scipy                                        1.10.1
sentence-transformers                        2.2.2
sentencepiece                                0.1.99
sentry-sdk                                   2.13.0
setproctitle                                 1.1.10
setuptools                                   70.3.0
sidetable                                    0.9.1
six                                          1.16.0
smart-open                                   5.2.1
smdebug-rulesconfig                          1.0.1
smmap                                        5.0.0
sniffio                                      1.3.0
snowflake-connector-python                   3.7.1
snowflake-snowpark-python                    1.11.1
sortedcontainers                             2.4.0
soupsieve                                    2.2.1
spacy                                        3.7.2
spacy-huggingface-pipelines                  0.0.4
spacy-legacy                                 3.0.12
spacy-loggers                                1.0.4
spacy-pkuseg                                 0.0.32
SQLAlchemy                                   2.0.23
SQLAlchemy-Utils                             0.41.2
sqlparse                                     0.5.0
srsly                                        2.4.6
sshpubkeys                                   3.3.1
stack-data                                   0.5.1
statsmodels                                  0.14.1
streamlit                                    1.37.1
SudachiDict-core                             20220519
SudachiPy                                    0.6.7
sympy                                        1.12
tblib                                        3.0.0
tenacity                                     8.2.2
tensorboard                                  2.16.2
tensorboard-data-server                      0.7.2
tensorflow                                   2.16.1
tensorflow-hub                               0.16.1
tensorflow-io-gcs-filesystem                 0.37.0
tensorflow-macos                             2.16.1
tensorflow-text                              2.16.1
termcolor                                    2.3.0
text-unidecode                               1.3
textwrap3                                    0.9.2
tf_keras                                     2.16.0
thefuzz                                      0.19.0
thinc                                        8.1.12
threadpoolctl                                3.2.0
tiktoken                                     0.7.0
tldextract                                   5.1.1
tokenizers                                   0.15.0
toml                                         0.10.2
tomlkit                                      0.11.8
toolz                                        0.12.0
torch                                        2.2.0
torchvision                                  0.17.0
tornado                                      6.4.1
tqdm                                         4.66.4
traitlets                                    5.9.0
transformers                                 4.39.3
typer                                        0.7.0
typing_extensions                            4.12.2
typing-inspect                               0.9.0
tzdata                                       2022.7
ulid-py                                      1.1.0
urllib3                                      1.26.19
wasabi                                       0.10.1
watchdog                                     3.0.0
wcwidth                                      0.2.5
weasel                                       0.3.4
websocket-client                             1.3.2
Werkzeug                                     3.0.3
wheel                                        0.40.0
Whoosh                                       2.7.4
widgetsnbextension                           4.0.10
wrapt                                        1.14.1
xarray                                       2023.1.0
xarray-einstats                              0.5.1
xmltodict                                    0.13.0
xxhash                                       2.0.2
yarl                                         1.9.2
zh_core_web_sm                               3.7.0
zhon                                         2.0.2
zipp                                         3.19.2

[RobertCraigie]

Thanks! I still can't reproduce with the dependencies you listed, I'm getting output like this:

RawMessageStartEvent(message=Message(id='msg_01Ez7uFLDPykQQReFiz1xwnK', content=[], model='claude-3-5-sonnet-20240620', role='assistant', stop_reason=None, stop_sequence=None, type='message', usage=Usage(input_tokens=366, output_tokens=10)), type='message_start')
RawContentBlockStartEvent(content_block=ToolUseBlock(id='toolu_01SbNhZSLyzi8m78Wp7QAbgc', input={}, name='get_weather', type='tool_use'), index=0, type='content_block_start')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='{"location"', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json=': "', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='San Franci', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockDeltaEvent(delta=InputJSONDelta(partial_json='sco, CA"}', type='input_json_delta'), index=0, type='content_block_delta')
RawContentBlockStopEvent(index=0, type='content_block_stop')
RawMessageDeltaEvent(delta=Delta(stop_reason='tool_use', stop_sequence=None), type='message_delta', usage=MessageDeltaUsage(output_tokens=41))
RawMessageStopEvent(type='message_stop')

Have you tested using the SDK outside of a notebook? I wonder if that's causing issues?

[kwnath]

Hmm interesting this is only happening on the notebook. I ran this in a python shell and I get multiple chunks.

[RobertCraigie]

Interesting, thanks. Could you share more details on how you have your notebook setup?

[kwnath]

Thanks @RobertCraigie I'll have a deeper look into how we're setting up our notebook but for now I'm happy to close this issue and update when I do find something in our setup!

[RobertCraigie]

Thanks for confirming! Did you manage to determine what was causing issues in your setup?

[RobertCraigie]

@kwnath did you manage to figure anything out?

[kwnath]

Hey Robert, I haven't yet, I had run a few tests around this and am still experiencing a few function calls without args. Will be investigating a bit more deeply today.

[kwnath]

@RobertCraigie I've done a bit more investigation just toggling on/off things in our app.

I finally pin pointed it to a specific package we initialise. Sentry seems to be the culprit. I'll make a reproducible example in a bit but it seems like they've had an issue before:

https://github.com/getsentry/sentry-python/releases/tag/2.10.0

Which links to this issue

I'm currently using 2.13 which should have fixed it but I think there is something else at play.

Edit:
Forgot to mention I'm on Python 3.11.7

[kwnath]

Okay here is how you can reproduce:

https://gist.github.com/kwnath/b5e432e7492c0ba9dec7740344d8891e

First try without initialising sentry, you'll notice the end output is there are tool args. Next initialise sentry and rerun the first example you'll notice there are no args.

[kwnath]

This is also problematic on the latest version of sentry:

langchain-anthropic==0.2.1
langchain-core==0.3.1
sentry-sdk==2.15.0

[kwnath]

I've figured this out! I'm going to write up a small report a bit later!

Making the fix here: getsentry/sentry-python#3615

I don't think there is much we can actually do here other than be wary about api changes in the future and use our own integrations.