epic: cortex models pull <ID_or_URL>

[Van-QA]

Goal

• Pull from Jan's curated models
• Pull from Huggingface based on URL

Tasklist

• feat: cortex pull supports full HF URL download #1004
• feat: Cortex pull from HF can select quantization #1006
• feat: Cache model and engine downloads so users can restart download #777

Bugs

• bug: cortex pull from hf times out #1017
• bug: Model corruption during download leads to loading failure #1139
• bug: cortex run or pull redownloads existing model multiple times #1344
• bug: Download models/engine failed cause api server stop #1010 (API server process crash edge case)
• #777

Out-of-scope

• Pause Functionality feat: Add pause functionality to model downloader jan#3519 (likely Sprint 21)
• Multi-mirror? (e.g. if HF down)
• Built in models to move back to Jan huggingface? (i.e. update URL)

[namchuai]

Updating a flow chart on fetch and downloading model from Hugging Face

[namchuai]

After downloading model, should we automatically check for engine and init engine (if neccessary)?

[dan-menlo]

@namchuai Can I check, does your implementation of cortex pull cover the case of a Huggingface URL to a specific GGUF model?

cortex pull https://huggingface.co/BafS/gemma-2-2b-it-Q4_K_M-GGUF/blob/main/gemma-2-2b-it-q4_k_m.gguf

[dan-menlo]

After downloading model, should we automatically check for engine and init engine (if neccessary)?

@namchuai From a UX perspective, I think we should not automatically init a model:

• We should show useful feedback to pre-empt problems
• e.g. "To run this model, you will need to initialize the TensorRT-LLM engine, i.e. cortex engines init tensorrt-llm

[namchuai]

@namchuai Can I check, does your implementation of cortex pull cover the case of a Huggingface URL to a specific GGUF model?

cortex pull https://huggingface.co/BafS/gemma-2-2b-it-Q4_K_M-GGUF/blob/main/gemma-2-2b-it-q4_k_m.gguf

Currently, we are not cover this case. This will be supported in upcoming release, I think.

[gabrielle-ong]

Marking as complete! 🎉