Does the MCP-Bridge support multiple inference endpoints ?

[dtrckd]

Hi,

I have a usecase where I can connect to different inference endpoints given their base_url/api_key.

Is it possible to pass multiple inference endpoints in the config.json?

Since we can bridge several MCP servers, I would like to bridge to different OpenAI API-compatible endpoints.
If not, could we pass the pair (base_url, api_key) at runtime to the completion routes?
If none of the above is possible, it seems that I would need to add another bridge in front of MCP-bridge to aggregate multiple OpenAI API-compatible endpoints, which feels like a lot of bridges...

[SecretiveShell]

This is not yet supported, though I think it's a good idea.

A possible roadmap for this would be:

a) rewrite the config schema to make the inference_server into inference_servers that is a list of inference_server with properties like

{
    "name": "example.com"
    "base_url": "http://example.com",
    "api_key": "abc"
}

b) rewrite the /v1/models endpoint to iterate over the list and asyncio.gather the results to form a complete list of all models and prefix the models

c) add model prefixing support and endpoint lookup to all the endpoints (though this can be implemented as a generic reuseable function)

Do you happen to have any explanation as to why you need this or what the use case is?

[dtrckd]

Sure, we are building an evaluation platform here: https://github.com/etalab-ia/eg1

When you design an experiment, you essentially provide a dataset, a model, and a set of metrics. A model can be any OpenAI-API compatible endpoint the user want to evaluate. I am currently working to integrate an MCP bridge so that it is possible to specify tooling when designing a model within an experiment.

I hope it makes sense.

[blevz]

+1 to model prefixing. My use case is I have ollama behind my mcp bridge instance and have a chat client configured against both ollama and mcp bridge and want to distinguish between the base models returned from ollama models endpoint and those returns from mcp-bridge's