bug: Engine Issues & API Issues

[louis-menlo]

Problem Statement

Backend

• There are a few differences between API documents and design, so:

  • How can I update the engine's settings, such as the API Key?

    Assuming users delete and add engines whenever they update them?

  • How can I know if there's an update available? When the backend checks for the new version, where does it notice the client? A data field, or a new endpoint? Currently, there's no additional field presented.

  • How can I update the engine versions? Will there be a new endpoint, or should I delete and re-add as above? What does this update do? How we distinguish between versions?How do we distinguish between versions? Does the remote provider provide that version name or we provision ourselves? It updates the models list, right?

  • How to update engine default request and response transform?

  • How to activate / deactivate an engine?

  • How can I get a list of models? How do I handle different parameters supported by each model?

    A new update merged yesterday, providing a list of models.

  • OS and hardware specifications constraint. Clients can know the constraints via /engines

  • How do we install a local engine? The local engine list is hardcoded, so we install variants, not engines. What data fields are needed if we planned to do that? For example, GitHub release link or local file path? How do we determine the required model fields for that engine? Does this engine have a specific model template when set up?

  • How to set engine display name?

    Assuming that can be added to metadata.

• The APIs should be polished to adhere to the standard RESTful API.

  • API Path:
    • Create action should post to the existing path, e.g., POST /v1/engines/{1}/install to create a new remote engine. This is incorrect because the entity {1} is unknown in the system. Some RESTful frameworks use upper decoration validation and reject this request. /v1/engines/ is a known path so that POST with body should work.
    • Follow CRUD, instead of introducing "/install" "/update", this would hard to maintain later when switching to other frameworks. - [ ] model.yml is cooked without aligned:

        "inference_params": {
             "temperature": 0.7,
             "stream": true
         },
        "TransformReq": {
            "get_models": {
                "url": "https://api.openai.com/v1/models"
             },
        }

  • Parameters & settings are grouped. Will this break the current version of the client? Has the backend handled and transformed it? What is the risk after the update?
  • Inconsistent between field names, snake case vs Pascal case
  • Some of those are optional but marked as required
  • engine | name vs engine_name, type vs without type

     "llama-cpp": [
      {
          "engine": "llama-cpp",
          "name": "mac-amd64",
          "version": "v0.1.42"
      },
      "openai": [
      {
         "engine_name": "openai",
         "type": "remote"

  • Choose Default Variant:
    • This implies that users can choose from various options of variants. I don't see it supported in model run

• Better error handling (not it hang without returning error message)

• Headers template instead of API Key Template

Frontend:

• How can we pre-populate the current supported engines to ensure compatibility with previous versions?
• How does the frontend decide which models to display, since /models returns a long list?
• Are there any clues for the supported parameters of those models?
• How do we define the default shipped variant? Since users can now download, there's no onboarding step for that.

Error handling

• There are different types of error structures. For instance, OpenRouter uses a streamed response, while others use a JSON response.

Resolved:

• - Use search query params in POST requests for variant defaults.

[louis-menlo]

@vansangpfiev can close the issue. Latest RC build already addressed those.