Multiple Function Calls in One Turn Fails to Accept Response

[TiVoShane]

Description

When utilizing a model that can call multiple function calls in one pass (i.e. gemini-1.5-flash-002), there is no way to provide multiple responses without a failure occurring.

In my example code, when asking about one exchange rate, I return one FunctionResponse and it succeeds. However, when asking about two exchange rates, and two FunctionCalls appear in one turn, I can't send a response without getting an error. I tried sending them as [FunctionResponse1, FunctionResponse2] and also separately, as two separate responses. Each time I get the following error:

11.4.0 - [FirebaseVertexAI][I-VTX002004] Response payload: {
"error": {
"code": 400,
"message": "Please ensure that function response turn comes immediately after a function call turn. And the number of function response parts should be equal to number of function call parts of the function call turn.",
"status": "INVALID_ARGUMENT"
}
}

Reproducing the issue

import Foundation
import FirebaseVertexAI

// Creating a TestAgent() will execute the testTwoFunctionCallsStreaming function.

class TestAgent {
var systemInstructions = """
Users will ask you information about exchange rates. Use your tools to answer them."
"""
private var modelName = "gemini-1.5-flash-002" // gemini-1.5-pro-exp-0801", //gemini-1.5-pro", //gemini-1.5-flash",
private var chat : Chat?
// Initialize the Vertex AI service
let vertex = VertexAI.vertexAI()
let config = GenerationConfig(
temperature: 0,
topP: 0.95,
topK: 40,
maxOutputTokens: 8192,
responseMIMEType: "text/plain"
)

init() {
    Task {
        await testTwoFunctionCallsStreaming()
    }
}

let getExchangeRate = FunctionDeclaration(
    name: "getExchangeRate",
    description: "Get the exchange rate for currencies between countries",
    parameters: [
        "currencyFrom": .string(
            description: "The currency to convert from."
        ),
        "currencyTo": .string(
            description: "The currency to convert to."
        ),
    ]
)

func makeAPIRequest(currencyFrom: String,
                    currencyTo: String) -> JSONObject {
    // This hypothetical API returns a JSON such as:
    // {"base":"USD","rates":{"SEK": 10.99}}
    var number = 10.99
    if currencyTo != "SEK" {
        number = 5.0
    }
    return [
        "base": .string(currencyFrom),
        "rates": .object([currencyTo: .number(number)]),
    ]
}

func testTwoFunctionCallsStreaming() async {
    print("testTwoFunctionCallsStreaming")
    let model = vertex.generativeModel(
        modelName: modelName,
        generationConfig: config,
        tools: [Tool.functionDeclarations( [getExchangeRate])],
        systemInstruction: ModelContent(role: "system", parts: self.systemInstructions)
    )
    // create some fake history.
    let mc1 = ModelContent(role: "user", parts: "Hello, how are you?")
    let mc2 = ModelContent(role: "model", parts: "I'm great!  How are you?")
    let mc3 = ModelContent(role: "user", parts: "I'm good.  Thanks for asking.")
    let mc4 = ModelContent(role: "model", parts: "My pleasure.  What can I do for you today?")
    
    let history = [mc1, mc2, mc3, mc4]
    chat = model.startChat(history: history)
    
    var response = try! chat?.sendMessageStream("Hello")
    await processResult(response: response!)
    
    response = try! chat?.sendMessageStream("What can you do?")
    await processResult(response: response!)

    response = try! chat?.sendMessageStream("How much is 50 US dollars worth in Swedish krona?")
    await processResult(response: response!)

    //Get one response to match the functioncall
    var apiResponse = makeAPIRequest(currencyFrom: "USD", currencyTo: "SEK")
   

    // Send the API response back to the model so it can generate a text response that can be
    // displayed to the user.
    response = try! chat?.sendMessageStream([ModelContent(
        role: "function",
        parts: [FunctionResponsePart(
            name: "getExchangeRate",
            response: apiResponse
        )]
    )])
    
    await processResult(response: response!)

    // Now ask it a question that will cause two function calls.
    response = try! chat?.sendMessageStream("How much is 50 US dollars worth in Swedish krona and the Euro?")
    await processResult(response: response!)

    //there should be two function calls.

    apiResponse = makeAPIRequest(currencyFrom: "USD", currencyTo: "SEK")
    let functionResponse1 = ModelContent(
        role: "function",
        parts: [FunctionResponsePart(
            name: "getExchangeRate",
            response: apiResponse
        )]
    )
    
    apiResponse = makeAPIRequest(currencyFrom: "USD", currencyTo: "EUR")
    let functionResponse2 = ModelContent(
        role: "function",
        parts: [FunctionResponsePart(
            name: "getExchangeRate",
            response: apiResponse
        )]
    )
    
    //METHOD 1
    // THIS FAILS
    let result = try! chat?.sendMessageStream([functionResponse1, functionResponse2])
    await processResult(response: result!)
    //  END METHOD 1
    
    
    //METHOD 2
    // THIS FAILS TOO

/*
let result = try! chat?.sendMessageStream([functionResponse1])
await processResult(response: result!)

    let result2 = try! chat?.sendMessageStream([functionResponse2])
    await processResult(response: result2!)

*/
// END METHOD 2
print(chat?.history.debugDescription ?? "No History")
print("END testTwoFunctionCalls")

}

func processResult(response : AsyncThrowingStream<GenerateContentResponse, Error>) async {
    do {
        for try await chunk in response {
            processResponseContent(content: chunk)
        }
    } catch {
    }
}

func processResponseContent(content: GenerateContentResponse) {
  guard let candidate = content.candidates.first else {
    fatalError("No candidate.")
  }

  for part in candidate.content.parts {
    switch part {
    case let textPart as TextPart:
      print(textPart.text)
     
    case let functionCallPart as FunctionCallPart:
        print(functionCallPart)
    default:
      fatalError("Unsupported response part: \(part)")
    }
  }
}

}

private extension [FunctionResponsePart] {
func modelContent() -> [ModelContent] {
return self.map { ModelContent(role: "function", parts: [$0]) }
}
}

Firebase SDK Version

11.4.0

Xcode Version

16.1

Installation Method

Swift Package Manager

Firebase Product(s)

VertexAI

Targeted Platforms

iOS

Relevant Log Output

11.4.0 - [FirebaseVertexAI][I-VTX002003] The server responded with an error: <NSHTTPURLResponse: 0x3017c8240> { URL: https://firebasevertexai.googleapis.com/v1beta/projects/expense-tracking-59dac/locations/us-central1/publishers/google/models/gemini-1.5-flash-002:streamGenerateContent?alt=sse } { Status Code: 400, Headers {
    "Alt-Svc" =     (
        "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
    );
    "Content-Length" =     (
        295
    );
    "Content-Type" =     (
        "text/event-stream"
    );
    Date =     (
        "Mon, 04 Nov 2024 17:06:07 GMT"
    );
    Server =     (
        ESF
    );
    Vary =     (
        Origin,
        "X-Origin",
        Referer
    );
    "x-content-type-options" =     (
        nosniff
    );
    "x-frame-options" =     (
        SAMEORIGIN
    );
    "x-xss-protection" =     (
        0
    );
} }
11.4.0 - [FirebaseVertexAI][I-VTX002004] Response payload: {
  "error": {
    "code": 400,
    "message": "Please ensure that function response turn comes immediately after a function call turn. And the number of function response parts should be equal to number of function call parts of the function call turn.",
    "status": "INVALID_ARGUMENT"
  }
}

If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet

Replace this line with the contents of your Package.resolved.

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet

Replace this line with the contents of your Podfile.lock!

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[andrewheard]

Hi @TiVoShane, it looks like in parallel function calling the backend requires that all of the function responses be in the same ModelContent (i.e., a single "function" turn with multiple parts). I see some potentially related, not yet released, validation changes in the backend but haven't looked in detail as to whether they loosen this specific requirement.

This modification to your code resolved the issue in my own testing (starting from //there should be two function calls and ending at // END METHOD 1):

var apiResponse = makeAPIRequest(currencyFrom: "USD", currencyTo: "SEK")
let functionResponse1 = FunctionResponsePart(name: "getExchangeRate", response: apiResponse)

apiResponse = makeAPIRequest(currencyFrom: "USD", currencyTo: "EUR")
let functionResponse2 = FunctionResponsePart(name: "getExchangeRate", response: apiResponse)

let result = try! chat?.sendMessageStream([
  ModelContent(role: "function", parts: functionResponse1, functionResponse2)
])
await processResult(response: result!)

Let me know if this unblocks you.

By the way, it was interesting to see how your prompt "What can you do?" seems to make the model more likely to accurately respond with a function call. I'd be curious to try adding its own response wording

I can access and process information from the available tools. Currently, I have access to a default_api which allows me to get exchange rates between different currencies. I can answer questions about exchange rates using this API. For example, you could ask me "What is the exchange rate from USD to EUR?".

or something similar to the system prompt to see if that accomplishes the same thing. Thanks for the tip!

[TiVoShane]

Interesting. Ok, then the example code in FunctionCallingViewModel would need to be altered. private extension [FunctionResponsePart] { func modelContent() -> [ModelContent] { return self.map { ModelContent(role: "function", parts: [$0]) } } } This code is creating multiple ModelContents. I altered the code to this and it's now working. private extension Array where Element == FunctionResponsePart { func modelContent() -> [ModelContent] { [ModelContent(role: "function", parts: self as [any Part])] } } Shane Miller https://twitter.com/TiVoShane https://www.instagram.com/tivoshane <https://www.instagram.com/tivoshane?igsh=OXV3amY0ejAzejgz&utm_source=qr>

…

On Mon, Nov 4, 2024 at 3:30 PM Andrew Heard ***@***.***> wrote: Hi @TiVoShane <https://github.com/TiVoShane>, it looks like in parallel function calling the backend requires that all of the function responses be in the same ModelContent (i.e., a single "function" turn with multiple parts). I see some potentially related, not yet released, validation changes in the backend but haven't looked in detail as to whether they loosen this specific requirement. This modification to your code resolved the issue in my own testing (starting from //there should be two function calls and ending at // END METHOD 1): var apiResponse = makeAPIRequest(currencyFrom: "USD", currencyTo: "SEK") let functionResponse1 = FunctionResponsePart(name: "getExchangeRate", response: apiResponse) apiResponse = makeAPIRequest(currencyFrom: "USD", currencyTo: "EUR") let functionResponse2 = FunctionResponsePart(name: "getExchangeRate", response: apiResponse) let result = try! chat?.sendMessageStream([ ModelContent(role: "function", parts: functionResponse1, functionResponse2) ]) await processResult(response: result!) Let me know if this unblocks you. By the way, it was interesting to see how your prompt "What can you do?" seems to make the model more likely to accurately respond with a function call. I'd be curious to try adding its own response wording I can access and process information from the available tools. Currently, I have access to a default_api which allows me to get exchange rates between different currencies. I can answer questions about exchange rates using this API. For example, you could ask me "What is the exchange rate from USD to EUR?". or something similar to the system prompt to see if that accomplishes the same thing. Thanks for the tip! — Reply to this email directly, view it on GitHub <#14022 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5ICKGI6T7ORECW37EDMALZ67KPBAVCNFSM6AAAAABREW5IJWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJVG42DCMJYGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[andrewheard]

Interesting. Ok, then the example code in FunctionCallingViewModel would need to be altered.

Good call, @TiVoShane. I've updated the extension in FunctionCallingViewModel with something similar now:

firebase-ios-sdk/FirebaseVertexAI/Sample/FunctionCallingSample/ViewModels/FunctionCallingViewModel.swift

Lines 251 to 255 in fa70701

	private extension [FunctionResponsePart] {
	func modelContent() -> ModelContent {
	return ModelContent(role: "function", parts: self)
	}
	}

It may also be more straightforward to omit the extension. I took that approach in our latest docs but either way works.

I'll close this now since it seems to be working for you but feel free to re-open if there are remaining concerns.