Assertion error when running EAGLE GPU example

[kyang-06]

Hi Team!

I followed this docs to run EAGLE, but encountered assertion error as shown in the picture.

ipex-llm/python/llm/example/GPU/Speculative-Decoding/EAGLE/README.md

Line 1 in 509bdb4

# EAGLE - Speculative Sampling using IPEX-LLM on Intel GPUs

I can confirmed that error dumped at the ipdb breakpoint: function layer_outputs = decoder_layer(...).

The execute script is

USE_XETLA=OFF SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 SYCL_CACHE_PERSISTENT=1 python -m evaluation.gen_ea_answer_llama2chat_e2_ipex_optimize \ 
--ea-model-path yuhuili/EAGLE-llama2-chat-7B \
--base-model-path meta-llama/Llama-2-7b-chat-hf \
--model-id e2-llama-2-7b-chat --enable-ipex-llm

My conda env is as below, also tried ipex-llm==2.1.0 which does not work:

bigdl-core-xe                    2.5.0b20231210
bigdl-core-xe-21                 2.6.0b20241212
bigdl-core-xe-addons-21          2.6.0b20241212
bigdl-core-xe-batch-21           2.6.0b20241212
bigdl-core-xe-esimd              2.5.0b20231210
bigdl-llm                        2.5.0b20231210
bitsandbytes                     0.41.2.post2
eagle-llm                        1.2.1                /data1/kyyx/github/LLM/speculative_decoding/EAGLE
huggingface-hub                  0.23.4
intel-cmplr-lib-ur               2024.2.0
intel-extension-for-pytorch      2.1.10+xpu
intel-extension-for-transformers 1.2.1
intel-openmp                     2024.2.0
ipex-llm                         2.2.0b20241212
torch                            2.1.0a0+cxx11.abi
torch-grammar                    0.3.3
torchvision                      0.16.0a0+cxx11.abi

As for eagle-llm package, I've tried install branch main or v1, both of them raise the same error.

[jenniew]

@kyang-06 Can you try earlier version of Eagle2 because recently there are some changes may affect the example code. You can try the older version like Jun 30 commit of main branch to see if there is any issue.

[kyang-06]

Jun 30 commit bffca4358a6615f9717ce82114d9a69ca9416091 still threw the error. VRAM uses 4.5GB, far away from OOM.

[kyang-06]

@jenniew Any update?