Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unexpected output when inference Qwen-7B-Chat-10-12 with 1024-128 in_out_pairs using transformer_int4_gpu api #9351

Open
WeiguangHan opened this issue Nov 3, 2023 · 2 comments
Open

unexpected output when inference Qwen-7B-Chat-10-12 with 1024-128 in_out_pairs using transformer_int4_gpu api #9351

WeiguangHan opened this issue Nov 3, 2023 · 2 comments

Comments

@WeiguangHan
Copy link
Contributor

@hkvision The output was about 41 which didn't meet expectations when I tested the Qwen-7B-Chat-10-1t model with 1024-128 in_out_pairs using transformer_int4_gpu api. Please have a look.

Env

bigdl-core-xe-2.4.0b20231102
bigdl-llm-2.4.0b20231102
intel-extension-for-pytorch-2.0.110+xpu
torch-2.0.1a0+cxx11.abi
torchvision-0.15.2a0+cxx11.abi
@hkvision
Copy link
Contributor

hkvision commented Nov 3, 2023

@qiuxin2012 Is it also due to the input prompt?

@qiuxin2012
Copy link
Contributor

@qiuxin2012 Is it also due to the input prompt?

Maybe, you can try some new input prompts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@qiuxin2012 @hkvision @WeiguangHan