How to build llama-cpp-python with Metal support?

[WojtekKowaluk]

I follow this: https://github.com/abetlen/llama-cpp-python#development
and this: https://github.com/abetlen/llama-cpp-python#installation-with-openblas--cublas--clblast
and this: https://github.com/ggerganov/llama.cpp#metal-build

git clone git clone --recurse-submodules [email protected]:abetlen/llama-cpp-python.git
cd llama-cpp-python
export LLAMA_METAL=1
export CMAKE_ARGS="-DLLAMA_METAL=on"
export FORCE_CMAKE=1
pip install .

However when loading model in text-generation-webui with n-gpu-layers set to 1 i get following errors:

INFO:Loading Wizard-Vicuna-7B_Uncensored...
INFO:llama.cpp weights detected: models/Wizard-Vicuna-7B_Uncensored/Wizard-Vicuna-7B-Uncensored.ggmlv3.q4_0.bin

INFO:Cache capacity is 0 bytes
llama.cpp: loading model from models/Wizard-Vicuna-7B_Uncensored/Wizard-Vicuna-7B-Uncensored.ggmlv3.q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =    0.07 MB
llama_model_load_internal: mem required  = 5407.71 MB (+ 1026.00 MB per state)
.
llama_init_from_file: kv self size  = 1024.00 MB
ggml_metal_init: allocating
ggml_metal_init: using MPS
ggml_metal_init: loading '(null)'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=258 "The file name is invalid."

Any idea what is wrong and how to fix it?

[WojtekKowaluk]

ok, found this topic: #317

[EricSimss]

Hi, could you provide a specific instruction on how you found the answer? What did you do, how do you know if metal is enabled or not.

Thank you!

[WojtekKowaluk]

this workaround works for me: #317 (comment)
but now I wait for proper solution

[ianscrivener]

OK, I officially give up... I tried every possible permutation and cannot get llama-cpp-python (v0.1.59) to build with or without GPU on MacOS M2. Here are my test reports;

1. https://gist.github.com/ianscrivener/6ebf0ffc6b2ba64042a035c110511229
2. https://gist.github.com/ianscrivener/5e9e5fef7de60f032df3a3854a6e3100
3. https://gist.github.com/ianscrivener/08e67f95245e823479de1a61068e462b
4. https://gist.github.com/ianscrivener/58339f2e6ff163588609d335f8477544
5. https://gist.github.com/ianscrivener/c9e281ffc790cd41991be3675dc72d7f
6. https://gist.github.com/ianscrivener/66a1915b7caecd219602b4a3e5d49684
7. https://gist.github.com/ianscrivener/6237c178b5b37198666f43c75b18f915

I can, however, get llama-cpp-python (v0.1.59) to install via standard pip ... albeit without Metal GPU support.

I had a new conda env, with python 3.9.16, tried ggml-metal.metal next to the pytohn executable etc etc etc..

[WojtekKowaluk]

Works now with 74fbaae