epic: New Model Support SOP + Systems

[dan-homebrew]

Goal

• Support new models within 24 hours of release in Jan and Cortex

Process

1. Create model conversion request ticket to Github board (janhq/jan or janhq/cortex.cpp) @gabrielle-ong

2. Convert Model @gabrielle-ong

   1. New model is published by author - Instruct models only. See license
   2. debug & curate model.yml @nguyenhoangthuan99: stop tokens, max_tokens, ngl, ctx_len, prompt_template
   3. Create HF cortexso repo, model card, with correct descriptions, attribution, etc
   4. From author's repo, ungate access if needed (logged in with team HF account for CI)
   5. Use Model Converter to convert the model into GGUF, TRTLLM, and ONNX, and upload to HF
   6. Add model.yml (no default)

3. QA (Cortex Team - @gabrielle-ong)
   6. Ensure it is downloadable and usable on Cortex
   8. Ensure it is downloadable and usable on Jan
   9. Debug if necessary

4. Marketing @eckartal
   11. Craft a mini update to users

Systems

• Repo janhq/models: Holds converter, and we hold "Model Inventory" there
  • Submodule?
• Kanban board for type: model requests, imported from janhq/jan and janhq/cortex.cpp model requests
  • https://github.com/orgs/janhq/projects/5/views/67
• HF: huggingface.co/janhq (no more cortexso)

Tasklist

• https://github.com/janhq/cortex.cpp/blob/dev/.github/ISSUE_TEMPLATE/model_request.yml
• create project board
• Automation to add Model Request Ticket to project board

Out-of-scope

• Jan Hub pulls from remote Model Hub

Related

• epic: Model Converter Pipeline #22
• roadmap: Built-in Model Library #21

[dan-homebrew]

I think we need to a process to support this:

• Repo: janhq/models (holds the converter scripts?)
• Project Board with Model Status (with clear ownership of each Status + Process)

There are really two roles:

• @eckartal to create ticket for new models
• @hahuyhoang411 to do basic conversion and testing
• Cortex team to step in only if it doesn't work directly out of the box

[gabrielle-ong]

@hahuyhoang411 thanks for taking this on and releasing the Models! Closing this issue