Reasoning models and structured output (e.g. QwQ-32B and JSON)

[matthew-at-qamcom]

I'm trying to understand how to run reasoning models (e.g. QwQ-32B) and also use structured output (specifically, I'm interested in JSON).

I have tried the examples in the documentation, and I'm very surprised by the output. For the first three examples, I see output via the reasoning_content and not on content. The fourth example I cannot get to run.

Unfortunately there is nothing on the website to say what the expected result is. Is this expected? I would have thought that you'd want the reasoning content (between the <think> and </think> tags) to be completely free-form and to enforce the structure only on the content. But that's not at all what I'm seeing. Do I misunderstand something?

Thanks in advance.

My output for the first example was:

reasoning_content:  London
content:  None

My output for the second example was:

reasoning_content:  {
  "name": "Alice",
  "age": 34
}
content:  None

For the first two examples, as per the documentation, I ran

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --enable-reasoning --reasoning-parser deepseek_r1

To get the third example to work, I had to run

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --enable-reasoning --reasoning-parser deepseek_r1 --guided-decoding-backend auto

The code for the first example is:

from enum import Enum

from openai import OpenAI
from pydantic import BaseModel

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

# Guided decoding by Regex
prompt = ("What is the capital of France?")

completion = client.chat.completions.create(
    model=model,
    messages=[{
        "role": "user",
        "content": prompt,
    }],
    extra_body={
        "guided_regex": "(Paris|London)",
    },
)
print("reasoning_content: ", completion.choices[0].message.reasoning_content)
print("content: ", completion.choices[0].message.content)

EDIT:
I'm using vllm 0.8.2

[matthew-at-qamcom]

What I would have expected with reasoning parser enabled, with structured output

For the first example, I was hoping I'd see output that was something like this:

reasoning_content:
Okay, so I need to figure out the capital of France. I remember that
France is a country in Europe, but I'm not exactly sure where its
capital is. Let me think about what I know. I know that the capital of
France is Paris, but I'm not 100% certain. I think Paris is the most
well-known city, but maybe there are other capitals too? Or is Paris
the official capital?

I should probably start by recalling the capital cities of different
countries. I know that the capital of France is Paris, but I also
remember that some countries have multiple capitals. For example, the
United States has Washington, D.C., and sometimes another city. Maybe
France is similar? But I think France only has one capital. Let me
think about other capitals I know. Like, London is the capital of the
UK, but that's not France. Maybe another city in France? I don't think
so.

Wait, I'm pretty sure Paris is the capital. But I should double-check
to make sure. Maybe I can think about the geography of France. Paris
is the capital, and it's the largest city in France. The rest of the
country is spread out, so there are other cities, but I don't think
they're capitals. So, I think Paris is the only capital of France.

I should also consider the history of France. I remember learning that
the French monarchy had a capital, but I think that's not the same as
the current capital. So, the current capital is still Paris, even with
the monarchy.

Another way to confirm is to think about the spelling of the word
"capital." Paris is the capital, so I can be pretty confident that's
the answer. I don't think there's any other city named Paris in
France, so that must be the official capital.

I guess the answer is straightforward: Paris. It's the capital of
France. I don't need to list other capitals because there aren't
any. So, the final answer should be Paris.

content:  
Paris

Reasoning parser enabled, without structured output

If I run with reasoning enabled using:

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --enable-reasoning --reasoning-parser deepseek_r1

And I comment out the extra_body part of the request (i.e. disable the structured out), then I get the result in the content section along with the reasoning in the reasoning_content section---all as I'd expect:

reasoning_content:
Okay, so I need to figure out the capital of France. I remember that
France is a country in Europe, but I'm not exactly sure where its
capital is. Let me think about what I know. I know that the capital of
France is Paris, but I'm not 100% certain. I think Paris is the most
well-known city, but maybe there are other capitals too? Or is Paris
the official capital?

I should probably start by recalling the capital cities of different
countries. I know that the capital of France is Paris, but I also
remember that some countries have multiple capitals. For example, the
United States has Washington, D.C., and sometimes another city. Maybe
France is similar? But I think France only has one capital. Let me
think about other capitals I know. Like, London is the capital of the
UK, but that's not France. Maybe another city in France? I don't think
so.

Wait, I'm pretty sure Paris is the capital. But I should double-check
to make sure. Maybe I can think about the geography of France. Paris
is the capital, and it's the largest city in France. The rest of the
country is spread out, so there are other cities, but I don't think
they're capitals. So, I think Paris is the only capital of France.

I should also consider the history of France. I remember learning that
the French monarchy had a capital, but I think that's not the same as
the current capital. So, the current capital is still Paris, even with
the monarchy.

Another way to confirm is to think about the spelling of the word
"capital." Paris is the capital, so I can be pretty confident that's
the answer. I don't think there's any other city named Paris in
France, so that must be the official capital.

I guess the answer is straightforward: Paris. It's the capital of
France. I don't need to list other capitals because there aren't
any. So, the final answer should be Paris.

content:  
The capital of France is Paris.

The problem with this result is the lack of structure in the response.

Reasoning parser disabled, with structured output

If I run without reasoning enabled using

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B

Then I get the result in the content section, as I'd expect:

reasoning_content:  None
content:  London

But in this scenario, I imagine all the reasoning of the model has been effectively turned off.

Reasoning parser disabled, without structured output

If I run without reasoning enabled using

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B

And I comment out the extra_body part of the request (i.e. disable the structured out), then I get the result in the content section along with the reasoning in the content section---all as I'd expect:

reasoning_content:  None
content: Okay, so I need to figure out the capital of France. Hmm,
where do I start? I remember that France is a country in Europe, but
I'm not exactly sure about its capital. I think it's somewhere along
the coast, maybe in the Mediterranean. I've heard terms like "Paris"
and "London" before, but I'm not sure which one is the actual capital.

Wait, I think Paris is the capital of France. I remember hearing that
Paris is a major city with lots of attractions, like the Eiffel Tower
and the Louvre Museum. But I also recall that London is the capital of
the United Kingdom. So maybe Paris is the capital of another country,
but the question is about France specifically.

Let me think about the spelling. France is spelled F-R-A-I-E-S. The
capital is the capital of France, so it's F-R-A-I-E-S-C-A-L. I think
the capital is Paris. I've heard that Paris is also the capital of
another country, like Canada, but that's not relevant here.

I should also consider the geography. Paris is in the northern part of
France, in the Île-de-France department. It's a major city with a lot
of historical sites and attractions. I'm pretty sure it's the
administrative capital, but sometimes it's called the capital because
it's the capital of the Île-de-France region.

Wait, but I also remember hearing about the Eiffel Tower and the
Louvre Museum, which are in Paris, but those are in France, not the
United Kingdom. So maybe the user is mixing up the capitals of
different countries. Paris is the capital of France, London is the
capital of the UK.

So, putting it all together, the capital of France is Paris. I think
that's correct because I've heard so much about it. It's a significant
city with a rich history and a lot of attractions for tourists. I
don't think it's London because London is the capital of the UK, and
the question is about France.  </think>

The capital of France is Paris. Paris is the administrative and
historical capital of the country, known for its rich history,
landmarks like the Eiffel Tower, and the Île-de-France
department. While London is the capital of the United Kingdom, the
question specifically asks about the capital of France, making Paris
the correct answer.

This is great, but clearly fails to output only either "London" or "Paris".

[matthew-at-qamcom]

Oh, it's a V0-only feature! See this discussion.

Running

VLLM_USE_V1=0 vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --enable-reasoning --reasoning-parser deepseek_r1

Now gives the output

reasoning_content: Okay, so I need to figure out the capital of
France. I'm not entirely sure, but I think I remember it being one of
the most populous countries in Europe. Let me start by recalling what
I know about France.

First, France is a country in Western Europe, right? It's known for
being a country with a long history. I think it's one of the oldest
nations in the world. Now, the capital of a country is the capital
city where the government is located, which is usually the most
important administrative center.

I remember that France's capital is Paris. I think it's the most
famous city in France and the capital of its region, the Seine
department. That makes sense because Paris has a lot of historical
significance, especially in the arts, literature, and music. The
Eiffel Tower is a big part of its iconic image, so that might be a key
reason why it's the capital.

I should check if there are any other capitals mentioned. Sometimes
countries have multiple capitals, especially smaller ones. But for
France, I think Paris is the main and only capital. The other capitals
mentioned in my knowledge are Toulouse and Créteil, which are in
different regions like the north and south of France,
respectively. These are more regional capitals, not the main
administrative one.

I also recall that Paris is the capital of the Seine River. The Seine
River is a major waterway in France, so that's another reason why it's
so important. It's used for navigation and transportation, and the
river's name is a common sight in Paris, like the Seine River Bridge
and the Eiffel Tower.

Are there any other notable places in Paris that are significant? The
Louvre Museum comes to mind, which is one of the largest art museums
in the world. It's located in the Île-de-France district, which is a
major part of Paris. That's a good reason to mention Paris as the
capital because it's a hub for art and culture.

I also think about the Eiffel Tower, which is the most famous landmark
in Paris. It's a symbol of France and has been featured in countless
films, statues, and buildings. The height of the tower is about 330
meters, making it one of the tallest structures in the world. That's
another reason why Paris is considered a major city.

I should also consider the political and economic significance of
Paris. As the capital, it's the administrative center, and the
government is responsible for everything in France. The political
system in France, which is a republic, is centered around the
political system of Paris, so that ties into why it's the capital.

In terms of geography, Paris is located in the northern part of
France, specifically in the Île-de-France region. It's a coastal city,
which is why it's connected to the Seine River. This makes it a prime
location for trade, navigation, and cultural activities.

I wonder if there are any other capitals or if Paris is unique in
France. I think France has only one capital, which is Paris. The other
cities, like Toulouse and Créteil, are more regional capitals, serving
as administrative centers for their respective departments. So, Paris
is the main administrative capital, and that's what's often referred
to as the capital of France.

I also recall that Paris is the capital of the Seine River, which is a
significant geographical feature. The river has been an important part
of France's history, and the capital is a central location for its
economy and culture.

In summary, considering all these points—its historical significance,
the Eiffel Tower, the Louvre Museum, the Seine River, and its
political and economic role—it's clear that Paris is the capital of
France. It's the most iconic city, the administrative center of the
Seine department, and a hub for art, culture, and trade.

content:  Paris