Skip to content

Custom URL support #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zachlatta opened this issue Mar 12, 2025 · 19 comments
Closed

Custom URL support #9

zachlatta opened this issue Mar 12, 2025 · 19 comments
Labels
enhancement New feature or request

Comments

@zachlatta
Copy link

First - wow! Thanks for making this!

Would love the ability to override the API base URL for any given model. I didn't see something in the docs around this.

This would enable support for Llama and proxies.
@crmne
Copy link
Owner

crmne commented Mar 14, 2025 •
edited
Loading

Hey @zachlatta - thanks for the kind words!

RubyLLM is designed around the idea that provider-specific details should be handled inside the library, not exposed as configuration. This gives us cleaner APIs and better handling of models than the "just swap the URL" approach.

We have an open issue for Ollama (#2), but we're implementing it as a proper provider rather than through OpenAI compatibility layer, which tends to be buggy and incomplete in my experience.

The same goes for other providers with OpenAI-compatible APIs - we'll add them as first-class providers in RubyLLM (like we did with DeepSeek) rather than making you fiddle with base URLs. This approach lets us properly handle model routing, capabilities, and pricing info.

If you're looking to use a specific provider, let us know which one and we can prioritize it. If it's about running models in your own infrastructure, that's valuable feedback for us to consider how to best support that use case.

@zachlatta
Copy link
Author

zachlatta commented Mar 14, 2025 via email

@jyunderwood
Copy link

This makes sense, but I'm not sure how you get around the need for configuring the url for provider like Azure OpenAI Service where your deployed models are at a unique endpoint for your resource.

We use Azure OpenAI at work for its unified billing and access control, and I would like to be able to use this library with our existing deployments.

@aspires
Copy link

aspires commented Mar 15, 2025

Hopping in here with another use case. Fastly's LLM semantic caching product, AI Accellerator, operates as a URL/API key swap within code to initiate middleware caching with Python and Node.js libraries.

Disclaimer: I work for Fastly. I'd love to feature a Ruby example within the product. If there's an easy drop-in integration option, we'll find a way to feature it.

@aspires aspires mentioned this issue Mar 16, 2025
Closed
@crmne
Copy link
Owner

crmne commented Mar 17, 2025

I still believe in our model-first approach rather than taking the "swap the URL in the config" shortcut. Each provider deserves proper first-class implementation with correct model routing, capabilities, and pricing info.

That said, Azure OpenAI presents a legitimate use case where URL configuration is inherent to the provider. I'll need to think about the right design to accommodate this without compromising our principles - maybe extending our OpenAI provider to handle Azure's deployment-based approach. Are pricing and capabilities the same as the original OpenAI models?

@jyunderwood
Copy link

There are probably enough differences in constructing the request that it would make sense for Azure OpenAI to be it's own provider.

However, the capabilities of Azure's hosted OpenAI APIs are the same as OpenAI's, and their pricing is identical, as far as I'm aware. For example, o3-mini costs $1.10 per million tokens on both platforms.

And the responses are the same structure.

Some differences:

Models can be deployed under custom names rather than their official model names. For instance, you could deploy the o3-mini model as "my-o3-mini," or whatever you want, and that would be the name used to access it istead of the official name.

The API key is sent as an api-key header when you use the static keys generated when your AI resource is created and as a bearer token using Authorization if you use Entra ID to get a per client or user token before sending the request.

Additionally, Azure requires an explicit API version. My understanding is this helps Azure stay in sync with OpenAI's API in terms of features and capabilities. So if new features are added to OpenAI's API, you would need to update the API version to gain access to those features.

An example Azure OpenAI request:

azure_api_key = "asdf-1234-asdf-1234"
azure_endpoint = "https://resource-name.openai.azure.com"
deployment_name = "my-deployment-name"  # Instead of official model name
api_version = "2024-10-21" # This is the latest GA version

# Instead of "https://api.openai.com/v1/chat/completions"
url = "#{azure_endpoint}/openai/deployments/#{deployment_name}/chat/completions?api-version=#{api_version}"

# Instead of { "Authorization" => "Bearer #{ENV['OPENAI_API_KEY']}" }
headers = {
  "api-key" => azure_api_key,
  "Content-Type" => "application/json"
}

# Note since the model is in the url, you don't need to put it in the body payload.
body = {
  "messages" => [{ "role" => "user", "content" => "Hello, how are you?" }]
}.to_json

response = HTTP.headers(headers).post(url, body: body)
puts JSON.pretty_generate(JSON.parse(response.body.to_s))

@crmne crmne added the enhancement New feature or request label Mar 23, 2025
@crmne
Copy link
Owner

crmne commented Mar 23, 2025

I appreciate the enthusiasm for RubyLLM, but I'm not particularly eager to turn this into a game of "config over code." URL overrides are essentially a backdoor that lets you bypass all the intelligence we've built into proper provider implementations.

The appeal of "just let me change the URL" is understandable, but it introduces a ton of implicit assumptions that I don't want in the codebase. When we implement a provider directly, we can ensure proper model routing, parameter handling, error management, and pricing information.

Azure OpenAI is a legitimate exception since their entire architecture is based on dynamic endpoints. We'll likely add proper Azure support in #15, but with a deliberate implementation, not by introducing a generic override.

For providers like LM Studio, Groq, or Ollama - I'd rather see purpose-built providers that correctly handle their quirks and capabilities than to have people hacking around with URL swaps.

@crmne crmne closed this as completed Mar 23, 2025
@howdymatt
Copy link

If it's about running models in your own infrastructure, that's valuable feedback for us to consider how to best support that use case.

I have this exact case. I plan to use LiteLLM to proxy our requests to models through an OpenAI-compatible interface. The underlying models will vary greatly. In my case, would I need to create a configuration for each application that uses the gateway, given that each application make end up using a set of models (they may want o3-mini and amazon-nova-pro)?

@crmne
Copy link
Owner

crmne commented Mar 26, 2025

@howdymatt if you're simply using an OpenAI interface for everything you should probably use @alexrudall's https://github.com/alexrudall/ruby-openai gem.

@zachlatta
Copy link
Author

zachlatta commented Mar 26, 2025 via email

@howdymatt
Copy link

@howdymatt if you're simply using an OpenAI interface for everything you should probably use @alexrudall's alexrudall/ruby-openai gem.

Got it. That is the library we use currently btw, I just like the idea of yours more :). Feels more Railsy, especially the tool calling functionality.

@crmne
Copy link
Owner

crmne commented Mar 26, 2025 •
edited
Loading

I hear you all on the need for more flexibility with custom endpoints and models. After some thinking, I think I've found a clean solution that maintains RubyLLM's principles while giving you the escape hatches you need.

Here's what we're going to do:

# Configure a custom endpoint if you need one
RubyLLM.configure do |config|
  config.openai_api_key = ENV['OPENAI_API_KEY']
  config.openai_api_base = "https://your-proxy.example.com"
end

# Default case - everything validated and proper
chat = RubyLLM.chat(model: "gpt-4")

# Need to use a custom model name? Just skip validation
chat = RubyLLM.chat(
  model: "gpt-9",
  provider: :openai,
  assume_model_exists: true
)

chat.with_model(model: "gpt-9", provider: :openai, assume_exists: true)

This handles pretty much every case that's come up:

  • @zachlatta - Your proxy case just works with openai_api_base
  • @jyunderwood - Azure deployments can use custom model names with assume_model_exists until we add proper Azure support in Azure OpenAI support #15
  • @howdymatt - LiteLLM proxying works the same way - use the custom URL and whatever model names your proxy exposes
  • @aspires - For Fastly's AI Accelerator, same deal - swap the URL and keep all the nice RubyLLM functionality

The assume_model_exists flag becomes particularly powerful when combined with provider - it lets you use any model ID with any provider, even if we don't know about it yet. This helps with new models, custom deployments, proxies, or any other case where you need to bypass our model registry.

I still believe in proper provider implementations as the best path (we'll keep adding official providers like we did with DeepSeek), but this gives you a clean way forward when you need to work outside those bounds.

How's that sound?

@crmne crmne reopened this Mar 26, 2025
@crmne
Copy link
Owner

crmne commented Mar 26, 2025 •
edited
Loading

The reason we need this and why I was a bit resistant to this change is because RubyLLM requires any model you use to be registered in our models.json file (or discoverable via models.refresh!). The assume_model_exists flag would let you bypass this requirement - particularly useful when:

# Using models we don't know about yet
chat = RubyLLM.chat(
  model: "gpt-5-preview",  # New model not in models.json
  provider: :openai,
  assume_model_exists: true
)

# Using custom deployment names
chat = RubyLLM.chat(
  model: "my-deployment",  # Custom name not in models.json
  provider: :openai,
  assume_model_exists: true
)

This gives you flexibility while keeping our default behavior safe and validated.

@howdymatt
Copy link

howdymatt commented Mar 26, 2025 •
edited
Loading

That works for my use case.

I assume that bypassing validation still allows us to use all the functionality (tool calling, etc) that exists for a given proxied model (as long as the proxy supports them as well)?

@crmne
Copy link
Owner

crmne commented Mar 26, 2025

It should work indeed.

@zachlatta
Copy link
Author

I love this proposal!

@gastonrey
Copy link

Hi team, is the skip_model_validation thing already implemented? I saw you created a PR for custom urls but then it was closed

@crmne
Copy link
Owner

crmne commented Apr 15, 2025 •
edited
Loading

Fixed in d628495 and 48f2faa

Grab the version from main and please test it!

The only change from the above message is that the option is assume_model_exists which is more ergonomic IMO, as is openai_api_base. I updated the above message.

@crmne crmne closed this as completed Apr 15, 2025
@crmne
Copy link
Owner

crmne commented Apr 15, 2025

https://rubyllm.com/guides/models#connecting-to-custom-endpoints--using-unlisted-models

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@crmne @aspires @zachlatta @jyunderwood @gastonrey @howdymatt