Skip to content

Commit

Permalink
Enable use_batch_forward Optimization on Battlemage GPU (#12516)
Browse files Browse the repository at this point in the history
* Update get_xpu_device_type() to support bmg

* enable use_batch_forward for bmg

* Update low_bit_linear.py

* Update utils.py

* use batch kernel for fp8e5
  • Loading branch information
liu-shaojun authored Dec 12, 2024
1 parent 6fc27da commit 2cce896
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 0 deletions.
1 change: 1 addition & 0 deletions python/llm/src/ipex_llm/transformers/low_bit_linear.py
Original file line number Diff line number Diff line change
Expand Up @@ -405,6 +405,7 @@ def use_batch_forward(x: torch.Tensor, qtype: int, output_len: int):
or (device in ["arc", "flex"] and qtype in [SYM_INT8, FP4])
or (device in ["arc", "flex", "mtl"] and qtype in [FP8E4])
or (device in ["lnl"] and qtype in [SYM_INT4] and x.shape[1] % 512 == 0)
or (device in ["bmg"] and qtype in [SYM_INT4, FP8E5])
)
return False

Expand Down
2 changes: 2 additions & 0 deletions python/llm/src/ipex_llm/transformers/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,8 @@ def get_xpu_device_type(x):
name = torch.xpu.get_device_name(x.device.index)
if name.startswith("Intel(R) Arc(TM) A"):
return "arc"
elif name.startswith("Intel(R) Graphics [0xe20b]"):
return "bmg"
elif name.startswith("Intel(R) Arc(TM)"):
if 'V' in name:
return "lnl"
Expand Down

0 comments on commit 2cce896

Please sign in to comment.