epic: SQLite database implementation

[nguyenhoangthuan99]

Implement a ModelListUtils class base on this discussion #1154

Overview

The ModelListUtils class provides a robust and thread-safe interface for managing a list of machine learning models. It handles operations such as adding, updating, deleting, and retrieving model information, with a focus on maintaining unique identifiers for each model.

Key Features

1. Thread-safe Operations: Uses mutex locking to ensure thread safety for all operations.
2. File-based Storage: Manages model information in a text file, allowing for persistence between program runs.
3. Unique Identifier Generation: Automatically generates unique, shortened aliases for models.
4. CRUD Operations: Supports Creating, Reading, Updating, and Deleting model entries.
5. Error Handling: Robust error checking and exception throwing for invalid operations.

Main Components

ModelEntry Struct

Represents a single model entry with fields:

• model_id: Unique identifier for the model
• author_repo_id: Author or repository identifier
• branch_name: Branch name in the repository
• path_to_model_yaml: Path to the model's YAML file
• model_alias: Shortened alias for the model
• status: Current status of the model (READY or RUNNING)

ModelListUtils Class Methods

1. LoadModelList()

   • Loads model entries from the file, creating the file if it doesn't exist.
   • Returns a vector of ModelEntry objects.

2. SaveModelList()

   • Saves the current list of model entries to the file.

3. GetModelInfo(identifier)

   • Retrieves model information based on model_id or model_alias.
   • Throws an exception if the model is not found.

4. AddModelEntry(new_entry, use_short_alias)

   • Adds a new model entry to the list.
   • Optionally generates a shortened alias for the model.
   • Ensures uniqueness of model_id and model_alias.

5. UpdateModelEntry(identifier, updated_entry)

   • Updates an existing model entry.
   • Identified by either model_id or model_alias.

6. DeleteModelEntry(identifier)

   • Deletes a model entry if it exists and is in READY state.

7. PrintModelInfo(entry)

   • Prints detailed information about a model entry.

8. GenerateShortenedAlias(model_id, entries)

   • Generates a unique, shortened alias for a model.
   • Uses a hierarchical approach to create aliases, prioritizing brevity while ensuring uniqueness.

Alias Generation Logic

The GenerateShortenedAlias function creates aliases in the following order:

1. Filename only (e.g., model_id_xxx)
2. Parent directory + filename (e.g., llama3.1-7b-gguf:model_id_xxx)
3. Grandparent directory + parent directory + filename (e.g., bartowski:llama3.1-7b-gguf/model_id_xxx)
4. Full path (e.g., huggingface.co:bartowski/llama3.1-7b-gguf/model_id_xxx)

It returns the shortest unique alias, appending a numeric suffix if necessary to ensure uniqueness.

[freelerobot]

Problem

• Cortex needs a model index to know which models are available (and if the model has been correctly configured)

Success Criteria

• models.list does not duplicate information in model.yaml
  • models.list has a clear separate purpose than the yaml.
  • duplicating fields across list and yaml introduces edge cases & unmaintainability
• cortex pull/run correctly updates models.list
• cortex import correctly updates models.list

Related

• feat: Cortex model importing #1058
• epic: Implement new Model Folder and model.yaml #1154

Questions

1. Naming: models.list or model.list? The former is more grammatically correct.
2. Why duplicate fields like author_repo_id, branch_name across both model.yaml and models.list? Should we move fields from model.yaml into list instead? OR vice versa?
3. Is there a better way to track model status than to do file IO during runtime?
4. Should we version the models.list? i.e. if we change the schema in the future, it is trackable.
5. Example: user symlinks a local model: llama3 local imported seems redundant

llama3 local imported /Users/nicolezhu/cortexcpp-nightly/models/imported/llama3.yml llama3 READY

6. @louis-jan mentioned using a db at one point. Does this beg that question?
7. When is model.list first created? Doesn't seem to be upon installation. Related: bug: Fail to get list model information: Unable to create model.list file #1285

Tldr

At the moment, the current models.list design is complicated. I am worried ab out the long term maintainability and edge cases.

1. Should model.list just be a bloody database?
2. What is the separation of concerns btw this and model.yaml (trying our best to not duplicate fields)

My 2cents:

• models.list: tracks model metadata needed for locally managing models, specific to cortex. e.g. id, alias, yaml path, maybe binary path
• model.yaml: tracks runtime parameters needed for running models, NOT specific to cortex runtime, e.g. source url, model name, all hyperparams...

cc @dan-homebrew

[freelerobot]

cc @louis-jan to take a look and lets discuss here 🙏

[nguyenhoangthuan99]

1. I'll update to models.list.
2. We can move all author_repo_id, branch_name to models.list
3. Currently, Sang has a command for checking model status by calling to server. We will remove the model status field. We also don't use IO read file for check model status currently.
4. We should version the models.list, I'll find the proper way to implement it.
5. We fill the models.list with this content llama3 local imported /Users/nicolezhu/cortexcpp-nightly/models/imported/llama3.yml llama3 READY because we are organizing models.list as a text file, so for easier to implement and avoid unexpected bugs, we must ensure the number of element in a line is constant.
   for example:

llama3 local imported /Users/nicolezhu/cortexcpp-nightly/models/imported/llama3.yml llama3 READY

mistral-model-id mistral/mistral-nemo 2407 /Users/nicolezhu/cortexcpp-nightly/models/huggingface.co/mistral/mistral-nemo/Mistral-Nemo.yml mistral-nemo-alias READY

This is our final model folder structure from this discussion #1154

/models
    model.list
    
    # Huggingface GGUF Model Folder format
    # This assumes we have some sort of quantization selection wizard when downloading?
    /huggingface.co
        /bartowski
            /Mixtral-8x22b-v0.1-gguf (includes all quants)
                Mixtral-8x22B-v0.1-IQ3_M.yaml
                Mixtral-8x22B-v0.1-IQ3_M-00001-of-00005.gguf...
                Mixtral-8x22B-v0.1-Q8_M.yaml
                Mixtral-8x22B-v0.1-Q8_M-00001-of-00005.gguf...

    # Built-in Library Model Folder format
    /cortex.so (this is our Built-in Model Library, based on Git, that will be mirrored across a few sites)
        /llama3.1 (model)
            /q4-tensorrt-llm (tag)
                ...engine_files
                model.yaml
            /q8-gguf **(tag)**
                model.yaml
   # Imported model from local 
    /imported
            <imported-model-id>.yaml

    # Future Model Source
    # Has its own model folder format

So we are aiming to make models.list can support model from many sources (local imported, cortexso, other hugging face branch, nvidia-ngc, ... ).

author_repo_id, branch_name also help us to know the source of model. We can use this information to show when list models.
6. I also think database for models.list instead of text file is necessary
7. Currently, models.list is first created as the first command cannot find models.list file. The issue #1285 caused by model pull command, still at old version and hasn't supported models.list yet. James and Sang is working with model run and model pull command with new models.list.

[freelerobot]

Gotcha, much of the above makes sense:

6. What's the reason for including the following in the model.list instead of yaml? (not saying its wrong, im just curious)

• author_repo_id: Author or repository identifier
• branch_name: Branch name in the repository

7. Should we just include the models.list in installation, as we write the cortexcpp folder? What do we get from dynamically creating it later on?

[nguyenhoangthuan99]

6. Model Metadata and Configuration:

• model.list: This is a header/metadata index containing information to identify models.
• model.yml: This is a configuration file that includes information for loading and running models. In the future, it may also be used for features like recommending inference parameters(ngl, context length) based on model specifications.
• author_repo_id and branch_name: These are not used for inference purposes. They are sometimes included in the name and model fields, but name and model primary purpose is to help the LlamaCpp engine distinguish between different models.

• We can add the necessary folder structure during the installation step. I'll collaborate with Hien on implementing this.
• Currently, both the engines and models folders are created automatically when they're not found at the first run.
• For development purposes, I need to create a models.list file to prevent crashes when running test cases in the Continuous Integration (CI) environment.

[freelerobot]

👍 all of the above makes sense, thank you for the clarification

…

On Mon, Sep 23, 2024 at 6:30 PM nguyenhoangthuan99 ***@***.***> wrote: 6. Model Metadata and Configuration: - model.list: This is a header/metadata index containing information to identify models. - model.yml: This is a configuration file that includes information for loading and running models. In the future, it may also be used for features like recommending inference parameters(ngl, context length) based on model specifications. - author_repo_id and branch_name: These are not used for inference purposes. They are sometimes included in the name and model fields, but name and model primary purpose is to help the LlamaCpp engine distinguish between different models. 7. - We can add the necessary folder structure during the installation step. I'll collaborate with Hien on implementing this. - Currently, both the engines and models folders are created automatically when they're not found at the first run. - For development purposes, I need to create a models.list file to prevent crashes when running test cases in the Continuous Integration (CI) environment. — Reply to this email directly, view it on GitHub <#1241 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQVWFCH7XQ5NOPFMVVNL46LZX7UTZAVCNFSM6AAAAABOMSC6AWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRXHAYTKMZUGQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[dan-menlo]

6. I also think database for models.list instead of text file is necessary

@0xSage @nguyenhoangthuan99 @louis-jan: If we are eventually going to move to a simple SQLite database for model management, I would prefer to do it now:

SQLite vs. models.list

• I think a simple SQLite database is better, if we take this approach
• We have a lot of moving parts depending on models.list, and a file corruption would brick both Jan and Cortex
• models.list fields are clearly structured, and SQL is better for this case

Versioning & Migration

• We would "version" the SQLite database by Cortex version (e.g. v0.1.0)
• Cortex Updater would run a database migration (up/down), vv @vansangpfiev
• We don't need to implement this for v0.1, but keep in mind as our strategy

cc @vansangpfiev @namchuai

[dan-menlo]

6. What's the reason for including the following in the model.list instead of yaml? (not saying its wrong, im just curious)

• author_repo_id: Author or repository identifier
• branch_name: Branch name in the repository

@nguyenhoangthuan99 @vansangpfiev @louis-jan @namchuai Let's start dealing with Model List as a SQLite database, and thinking through clearly what we need for Cortex to function.

I also see SQLite database would also allow us to track things like status, which is useful for operations.

• model_id: Unique identifier for the model
• author_repo_id: Author or repository identifier
• branch_name: Branch name in the repository
• path_to_model_yaml: Path to the model's YAML file
• model_alias: Shortened alias for the model
• status: Current status of the model (READY or RUNNING)

However, I note in our discussion, the example seems to have a local, imported in the struct.

• Are we also tracking "origin" of model?
• I think this could be very useful, especially for symlinked models

As part of this discussion exercise, can we do a few mockup entries, to see whether our assumptions work?

• Huggingface model import (e.g. https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GGUF)
• Cortex.so Model Hub import (i.e. openhermes2.5:7b)
• "Imported" or Symlinked model (i.e. sitting in a Kobold folder)

model_id	author_repo_id	branch_name	path_to_model_yaml	model_alias
	Cell	Cell	Cell	Cell
Cell	Cell	Cell	Cell	Cell

[namchuai]

IMO, I'm happy with using database approach. File-based approach does not play well with migration.
Couple of thoughts:

• Have to change cortex code base quite a lot.
• cortex update(maybe even downgrade) might need to update database schema as well?

[vansangpfiev]

I agree with the database approach. @nguyenhoangthuan99 implemented a good interface for model.list, so only need to change the implementation to replace it with database.

[gabrielle-ong]

Closing as complete - sqlite