[torch 2.3 + bigdl-core-xe-23] AttributeError: module 'xe_linear' has no attribute 'forward_qkv'

[Nuullll]

Environment

OS: Windows
Running Mistral-7B-Instruct-v0.3 using the example script generate.py with ipex-2.3 + ipex-llm-2.2 + bigdl-core-xe-*-23 on MTL iGPU.
I believe this issue can be reproduced with ACM dGPU (and other Arc graphics) too.

Reproduce steps

conda create -p env_bad python=3.11 libuv -y
conda activate .\env_bad

pip install torch==2.3.1.post0+cxx11.abi torchvision==0.18.1.post0+cxx11.abi torchaudio==2.3.1.post0+cxx11.abi intel-extension-for-pytorch==2.3.110.post0+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/mtl/us/

pip install --pre ipex_llm==2.2.0b2 bigdl-core-xe-23==2.6.0b2 bigdl-core-xe-addons-23==2.6.0b2 bigdl-core-xe-batch-23==2.6.0b2 onednn-devel==2024.1.1 transformers==4.39.0 accelerate trl sentence_transformers sentencepiece protobuf

python generate.py --repo-id-or-model-path <my_model_path>

##############

D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\transformers\deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: 'Could not find module 'D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torchvision\image.pyd' (or one of its dependencies). Try using the full path with constructor syntax.'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
2024-12-05 13:47:37,013 - INFO - intel_extension_for_pytorch auto imported
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 11.56it/s]
2024-12-05 13:47:37,712 - INFO - Converting the current model to sym_int4 format......
2024-12-05 13:47:38,350 - INFO - PyTorch version 2.3.1.post0+cxx11.abi available.
You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Traceback (most recent call last):
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\mistral.py", line 60, in <module>
    output = model.generate(input_ids,
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\ipex_llm\transformers\lookup.py", line 123, in generate
    return original_generate(self,
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\ipex_llm\transformers\speculative.py", line 109, in generate
    return original_generate(self,
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\ipex_llm\transformers\pipeline_parallel.py", line 281, in generate
    return original_generate(self,
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\transformers\generation\utils.py", line 1527, in generate
    result = self._greedy_search(
             ^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\transformers\generation\utils.py", line 2411, in _greedy_search
    outputs = self(
              ^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 1157, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\ipex_llm\transformers\models\mistral.py", line 217, in mistral_model_forward_4_36
    return MistralModel.forward(
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 1042, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 757, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\ipex_llm\transformers\models\mistral.py", line 1144, in mistral_attention_forward_4_39
    return forward_function(
           ^^^^^^^^^^^^^^^^^
  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\env_bad\Lib\site-packages\ipex_llm\transformers\models\mistral.py", line 1194, in mistral_attention_forward_4_39_original
    query_states, key_states, value_states = xe_linear.forward_qkv(hidden_states,
                                             ^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'xe_linear' has no attribute 'forward_qkv'. Did you mean: 'forward_new'?

Investigation notes

The issue does not exist for IPEX-2.1 + bigdl-core-xe-*-21:

conda create -p env_good python=3.11 libuv -y
conda activate .\env_good

pip install torch==2.1.0.post3 torchvision==0.16.0.post3 torchaudio==2.1.0.post3 intel-extension-for-pytorch==2.1.40+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/mtl/us/
pip install dpcpp-cpp-rt==2024.2 mkl-dpcpp==2024.2

pip install --pre ipex_llm==2.2.0b2 bigdl-core-xe-21==2.6.0b2 bigdl-core-xe-addons-21==2.6.0b2 bigdl-core-xe-batch-21==2.6.0b2 onednn-devel==2024.1.1 transformers==4.39.0 accelerate trl sentence_transformers sentencepiece protobuf

python generate.py --repo-id-or-model-path <my_model_path>

###########

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.3 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "D:\iusers\yilonggu\repro\ipex-llm-mistral\mistral.py", line 17, in <module>
    import torch
  File "D:\iusers\yilonggu\repro\env_good\Lib\site-packages\torch\__init__.py", line 1382, in <module>
    from .functional import *  # noqa: F403
  File "D:\iusers\yilonggu\repro\env_good\Lib\site-packages\torch\functional.py", line 7, in <module>
    import torch.nn.functional as F
  File "D:\iusers\yilonggu\repro\env_good\Lib\site-packages\torch\nn\__init__.py", line 1, in <module>
    from .modules import *  # noqa: F403
  File "D:\iusers\yilonggu\repro\env_good\Lib\site-packages\torch\nn\modules\__init__.py", line 35, in <module>
    from .transformer import TransformerEncoder, TransformerDecoder, \
  File "D:\iusers\yilonggu\repro\env_good\Lib\site-packages\torch\nn\modules\transformer.py", line 20, in <module>
    device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
D:\iusers\yilonggu\repro\env_good\Lib\site-packages\torch\nn\modules\transformer.py:20: UserWarning: Failed to initialize NumPy:
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.3 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

 (Triggered internally at C:\Jenkins\workspace\IPEX-GPU-ARC770-Windows-Build@4\frameworks.ai.pytorch.private-gpu\torch\csrc\utils\tensor_numpy.cpp:84.)
  device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
D:\iusers\yilonggu\repro\env_good\Lib\site-packages\transformers\deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
D:\iusers\yilonggu\repro\env_good\Lib\site-packages\torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: 'Could not find module 'D:\iusers\yilonggu\repro\env_good\Lib\site-packages\torchvision\image.pyd' (or one of its dependencies). Try using the full path with constructor syntax.'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
2024-12-05 13:55:56,100 - INFO - intel_extension_for_pytorch auto imported
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 12.91it/s]
2024-12-05 13:55:56,897 - INFO - Converting the current model to sym_int4 format......
2024-12-05 13:56:00,390 - INFO - PyTorch version 2.1.0.post3+cxx11.abi available.
You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Inference time: 1.7521936893463135 s
-------------------- Output --------------------
 What is AI?  Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think and learn like humans. It involves the development of computer

Related issue

The issue is exposed by Intel AI Playground: intel/AI-Playground#94

[Nuullll]

Got a workaround by disabling fast decoding path to avoid calling forward_qkv(): intel/AI-Playground@62920f6

[Oscilloscope98]

Hi @Nuullll,

This is a known issue for IPEX-LLM PyTorch 2.3 support on Mistral. We will fix it and let you know for any updates :)