Intel(R) Core(TM) Ultra 5 125H NPU so slowly?

[mnlife]

when I using Intel(R) Core(TM) Ultra 5 125H to test, npu is so slowly?

install npu driver follow this: https://github.com/intel/linux-npu-driver/blob/main/docs/overview.md

pip install optimum-intel nncf==2.11 onnx==1.16.1
pip install --pre openvino openvino-tokenizers openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

this is benchmark result

(envv) mnlife@ubuntu:~/openvino.genai/samples/python/benchmark_genai$ python benchmark_genai.py -p "why the Sun is yellow?" -nw 1 -n 1 -mt 200 -d CPU -m /home/mnlife/models/TinyLlama
Load time: 620.00 ms
Generate time: 70.05 ± 0.00 ms
Tokenization time: 0.38 ± 0.00 ms
Detokenization time: 0.55 ± 0.00 ms
TTFT: 69.48 ± 0.00 ms
TPOT: 69.48 ± 0.00 ms
Throughput : 14.39 ± 0.00 tokens/s
(envv) mnlife@ubuntu:~/openvino.genai/samples/python/benchmark_genai$ python benchmark_genai.py -p "why the Sun is yellow?" -nw 1 -n 1 -mt 200 -d NPU -m /home/mnlife/models/TinyLlama
Load time: 3325.00 ms
Generate time: 53633.71 ± 0.00 ms
Tokenization time: 0.36 ± 0.00 ms
Detokenization time: 0.50 ± 0.00 ms
TTFT: 4742.35 ± 0.00 ms
TPOT: 268.16 ± 317.23 ms
Throughput : 3.73 ± 4.41 tokens/s

[mnlife]

very slow when I using npu

(envv) mnlife@ubuntu:~/models$ time python genai.py CPU
args: CPU
it is the color of the sky.



real	0m1.636s
user	0m3.346s
sys	0m1.512s
(envv) mnlife@ubuntu:~/models$ time python genai.py NPU
args: NPU
it is the color of the sky.



real	0m11.004s
user	1m36.290s
sys	0m5.256s
(envv) mnlife@ubuntu:~/models$ cat genai.py 
import openvino_genai as ov_genai
import sys
print(f"args: {sys.argv[1]}")
pipe = ov_genai.LLMPipeline("TinyLlama", sys.argv[1])
config = ov_genai.GenerationConfig()
config.max_new_tokens = 10
prompt = "The Sky is blue because"
res = pipe.generate(prompt, config)
print(res)
(envv) mnlife@ubuntu:~/models$ 

[dmatveev]

very slow when I using npu

(envv) mnlife@ubuntu:~/models$ time python genai.py CPU
args: CPU
it is the color of the sky.



real	0m1.636s
user	0m3.346s
sys	0m1.512s
(envv) mnlife@ubuntu:~/models$ time python genai.py NPU
args: NPU
it is the color of the sky.



real	0m11.004s
user	1m36.290s
sys	0m5.256s
(envv) mnlife@ubuntu:~/models$ cat genai.py 
import openvino_genai as ov_genai
import sys
print(f"args: {sys.argv[1]}")
pipe = ov_genai.LLMPipeline("TinyLlama", sys.argv[1])
config = ov_genai.GenerationConfig()
config.max_new_tokens = 10
prompt = "The Sky is blue because"
res = pipe.generate(prompt, config)
print(res)
(envv) mnlife@ubuntu:~/models$ 

Hi @mnlife , note - when you measure time end-to-end, you also include the model compile time on NPU - so it may show higher overall number than for CPU and GPU.

Please wait the new OpenVINO release and driver update, I hope @TolyaTalamanov will help you get better results.

[mnlife]

very slow when I using npu

(envv) mnlife@ubuntu:~/models$ time python genai.py CPU
args: CPU
it is the color of the sky.



real	0m1.636s
user	0m3.346s
sys	0m1.512s
(envv) mnlife@ubuntu:~/models$ time python genai.py NPU
args: NPU
it is the color of the sky.



real	0m11.004s
user	1m36.290s
sys	0m5.256s
(envv) mnlife@ubuntu:~/models$ cat genai.py 
import openvino_genai as ov_genai
import sys
print(f"args: {sys.argv[1]}")
pipe = ov_genai.LLMPipeline("TinyLlama", sys.argv[1])
config = ov_genai.GenerationConfig()
config.max_new_tokens = 10
prompt = "The Sky is blue because"
res = pipe.generate(prompt, config)
print(res)
(envv) mnlife@ubuntu:~/models$ 

Hi @mnlife , note - when you measure time end-to-end, you also include the model compile time on NPU - so it may show higher overall number than for CPU and GPU.

Please wait the new OpenVINO release and driver update, I hope @TolyaTalamanov will help you get better results.

Very much looking forward to your reply

[junruizh2021]

@dmatveev Does performance of LLM on NPU rely on "remote tensors" feature? I also observed that the performance on NPU is worse than CPU.

[mnlife]

very slow when I using npu

(envv) mnlife@ubuntu:~/models$ time python genai.py CPU
args: CPU
it is the color of the sky.



real	0m1.636s
user	0m3.346s
sys	0m1.512s
(envv) mnlife@ubuntu:~/models$ time python genai.py NPU
args: NPU
it is the color of the sky.



real	0m11.004s
user	1m36.290s
sys	0m5.256s
(envv) mnlife@ubuntu:~/models$ cat genai.py 
import openvino_genai as ov_genai
import sys
print(f"args: {sys.argv[1]}")
pipe = ov_genai.LLMPipeline("TinyLlama", sys.argv[1])
config = ov_genai.GenerationConfig()
config.max_new_tokens = 10
prompt = "The Sky is blue because"
res = pipe.generate(prompt, config)
print(res)
(envv) mnlife@ubuntu:~/models$ 

Hi @mnlife , note - when you measure time end-to-end, you also include the model compile time on NPU - so it may show higher overall number than for CPU and GPU.

Please wait the new OpenVINO release and driver update, I hope @TolyaTalamanov will help you get better results.

Hi, Has this issue been resolved?