Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Chatglm2 rope optimization on xpu #9350

Merged
merged 17 commits into from
Nov 6, 2023
Merged

Chatglm2 rope optimization on xpu #9350

merged 17 commits into from
Nov 6, 2023

Conversation

qiuxin2012
Copy link
Contributor

@qiuxin2012 qiuxin2012 commented Nov 3, 2023 •
edited
Loading

Description

Speed up chat glm2 on xpu

1. Why the change?

Speed up chat glm2 on xpu with ipex

2. Summary of the change

  1. use torch.ops.torch_ipex.apply_rotary_embedding instead of apply_rotary_pos_emb
  2. (sin, cos)'s repeat_interleave in model's forward if xpu is used. (cpu unchanged)

@qiuxin2012 qiuxin2012 requested a review from yangw1234 November 3, 2023 12:14
@qiuxin2012 qiuxin2012 assigned qiuxin2012 and unassigned qiuxin2012 Nov 6, 2023
@qiuxin2012 qiuxin2012 requested a review from hkvision November 6, 2023 02:34
hkvision
hkvision reviewed Nov 6, 2023
View reviewed changes
python/llm/src/bigdl/llm/transformers/models/chatglm2.py
Comment on lines +122 to +123
use_fuse_rope = input_ids.device.type == "xpu"
use_fuse_rope = use_fuse_rope and not self.training
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can combine this to if input_ids.device.type == "xpu" and not self.training

@hkvision hkvision mentioned this pull request Nov 6, 2023
Closed
@qiuxin2012
Copy link
Contributor Author

Upstream output:

Once upon a time, there was a young girl named Samantha who lived with her parents in a small town. Samantha had always dreamed of traveling the world and experiencing new cultures. One day, she heard about a travel agency that offered guided tours to far-off lands.

The travel agency offered a variety

PR's output:

Once upon a time, there was a young girl named Samantha who lived with her parents in a small town. Samantha had always dreamed of traveling the world and experiencing new cultures. One day, she heard about a travel agency that offered guided tours to far-off lands.

The travel agency offered a variety

@qiuxin2012 qiuxin2012 merged commit 0cd751f into intel-analytics:main Nov 6, 2023
23 checks passed
@jason-dai
Copy link
Contributor

Need to add test similar to #9347

@qiuxin2012
Copy link
Contributor Author

Need to add test similar to #9347

Correctness test in #9450

liu-shaojun pushed a commit that referenced this pull request Mar 25, 2024
@qiuxin2012
Chatglm2 rope optimization on xpu (#9350)
1420e45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hkvision hkvision hkvision approved these changes

@yangw1234 yangw1234 Awaiting requested review from yangw1234

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@qiuxin2012 @jason-dai @hkvision