How to Use Metal on Intel Mac?

[Spider-netizen]

Hi,

I'm trying to run a model using Metal on an intel Mac. I'm getting an error. This is the end of the traceback:

Does anyone happen to know what I am missing?

[thiner]

I think "Metal" refers to M Series chip, which is ARM architecture, that's not built for Intel chip. You should try the CPU version, but that would be extremely slow in my opinion.

[andreped]

@thiner's answer is correct.

But for anyone wondering how to use Llama3 on Metal for ARM chip, you can install using:

CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python

Then perform completion using this example (here I used the Q2 quantized Llama3 variant:

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="QuantFactory/Meta-Llama-3-8B-Instruct-GGUF",
    filename="Meta-Llama-3-8B-Instruct.Q2_K.gguf",
    verbose=True,
    n_gpu_layers=-1,
)

output = llm(
      "Q: List the planet names in our solar system? A: ",
      max_tokens=32,
      stop=["Q:", "\n"],
      echo=True,
)

[Spider-netizen]

To my understanding Metal utilises mac's gpus for better performance. On the official website it says it supports my computer. Also llama.cpp can be build with Metal on my computer as it seems.

…

On Tue, 23 Apr 2024, 07:07 Sebastian.W, ***@***.***> wrote: I think "Metal" refers to M Series chip, which is ARM architecture, that's not built for Intel chip. You should try the CPU version, but that would be extremely slow in my opinion. — Reply to this email directly, view it on GitHub <#1365 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL33HPR7XERIC3T5V6F2VULY6XNBFAVCNFSM6AAAAABGPTTZV6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCOJWGA2TE> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[thiner]

To my understanding Metal utilises mac's gpus for better performance.

Yes, it utilize Apple GPU, and Apple GPU is available only with M series chips.