roadmap: Jan has Hardware Controls and System Monitor and Prioritization

[dan-menlo]

Goal

• Migrate Jan's Monitoring and Hardware Compatibility APIs to Cortex
  • e.g. System Monitor
  • e.g. Model Compatibility

Out-of-scope

• AMD GPUs
• Intel GPUs

Tasklist

Issues

• roadmap: Jan has Hardware Controls and System Monitor and Prioritization #3908

Frontend

• Designs & UI Engineering: feat: Implement UI for Better Hardware Settings  #3688
  • Prioritization of Nvidia hardware @imtuyethan [0,1,2] || [id1, id2, id3]

Backend

• planning: Cortex Hardware API cortex.cpp#1165
• epic: Implement Cortex Hardware API for Nvidia cortex.cpp#1568
• planning: Cortex Model Compatibility API cortex.cpp#1108
• planning: prioritize GPUs with CUDA_VISIBLE_DEVICES cortex.cpp#1679
• bug: some models failed to load if many GPU are selected cortex.cpp#1458
• bug: insufficient handling of insufficient memory cortex.cpp#1457
• roadmap: Jan & Cortex support for AMD GPUs  #3529

[dan-menlo]

12 Dec

Hardware UI

Open Questions

• How do we prioritize across both Nvidia and AMD GPUs? @vansangpfiev
• UX for Vulkan (as it can run on both Nvidia and AMD)
• Should we have separate settings for llama.cpp Engines and Hardware (eg. for Vulkan)

[vansangpfiev]

How do we prioritize across both Nvidia and AMD GPUs?

AMD and Nvidia have uuid field, we can use this field to map between GPU and the index we pass to CUDA_VISIBLE_DEVICES and GGML_VULKAN_DEVICE env variable.

[dan-menlo]

@vansangpfiev Please add in architecture diagrams and Engineering Spec

[vansangpfiev]

• Diagram to configure global GPUs prioritization
• Vulkan can detect both AMD GPUs and Nvidia GPUs
• Nvidia-smi can only detect Nvidia GPUs
• We have a mapping function to map uuid with cuda index and vulkan index

cc: @dan-homebrew @louis-jan