Skip to content

[UI]: Support in-server model proxy #1954

Closed
@jvstme

Description

@jvstme

Context

#1595 adds support for running models without a gateway, in which case the model is served by dstack-server and the model URL consists only of a dstack-server path. Frontend changes are required to correctly display such URLs on the Models page (/models) and use them with the OpenAI client in the model chat.

Current behavior

  • The table on the Models page has a column named "Gateway".
  • The service.model.base_url field returned by /api/runs/get or /api/runs/list always contains a full URL that is then displayed in the "Gateway" column and passed to the OpenAI client as is.

New expected behavior

  • The column should be renamed "URL".
  • The service.model.base_url field may contain either a full URL or only a path.
    • If it's a full URL, it should be displayed in the "URL" column and passed to the OpenAI client as is.
    • If it's a path, the URL displayed in the "URL" column and passed to the OpenAI client should be constructed by adding that path to the dstack-server base URL. 1

Examples

dstack-server base URL The service.model.base_url field Resulting model URL
any https://gateway.admin.sky.dstack.ai https://gateway.admin.sky.dstack.ai
any http://gateway.mycompany.example http://gateway.mycompany.example
https://sky.dstack.ai /proxy/models/admin/ https://sky.dstack.ai/proxy/models/admin/
http://localhost:3000 /proxy/models/admin/ http://localhost:3000/proxy/models/admin/

Footnotes

  1. The dstack-server base URL here means the scheme, hostname, and optional port. The /api prefix is not part of the base URL.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions