Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable use_batch_forward Optimization on Battlemage GPU #12516

Merged
merged 5 commits into from
Dec 12, 2024
Merged

Enable use_batch_forward Optimization on Battlemage GPU #12516

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3a72f5f
Update get_xpu_device_type() to support bmg
liu-shaojun Dec 9, 2024
b696335
enable use_batch_forward for bmg
liu-shaojun Dec 9, 2024
56247e6
Update low_bit_linear.py
liu-shaojun Dec 9, 2024
478fcdf
Update utils.py
liu-shaojun Dec 9, 2024
c0225e9
use batch kernel for fp8e5
liu-shaojun Dec 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions python/llm/src/ipex_llm/transformers/low_bit_linear.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -405,6 +405,7 @@ def use_batch_forward(x: torch.Tensor, qtype: int, output_len: int):
or (device in ["arc", "flex"] and qtype in [SYM_INT8, FP4])
or (device in ["arc", "flex", "mtl"] and qtype in [FP8E4])
or (device in ["lnl"] and qtype in [SYM_INT4] and x.shape[1] % 512 == 0)
or (device in ["bmg"] and qtype in [SYM_INT4])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To support more inference and serving requirements on BMG, shall we also verify that FP8/INT8 can benefit from use_batch_forward. @jason-dai

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I collected FP8 data on BMG using the all-in-one. Models larger than 7B cause the machine to crash, while models smaller than 7B show a 1-2% improvement in next-token performance after enabling the batch kernel.

Should we enable the batch kernel for qtype=FP8E5 as well?

)
return False

Expand Down
2 changes: 2 additions & 0 deletions python/llm/src/ipex_llm/transformers/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,8 @@ def get_xpu_device_type(x):
name = torch.xpu.get_device_name(x.device.index)
if name.startswith("Intel(R) Arc(TM) A"):
return "arc"
elif name.startswith("Intel(R) Graphics [0xe20b]"):
return "bmg"
elif name.startswith("Intel(R) Arc(TM)"):
if 'V' in name:
return "lnl"
Expand Down
Loading