Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] "topk_cpu" not implemented for 'Half' #31

Open
Lyaaaaaaaaaaaaaaa opened this issue May 8, 2024 · 1 comment
Open

[BUG] "topk_cpu" not implemented for 'Half' #31

Lyaaaaaaaaaaaaaaa opened this issue May 8, 2024 · 1 comment
Assignees
@Lyaaaaaaaaaaaaaaa
Labels
bug Something isn't working

Comments

@Lyaaaaaaaaaaaaaaa
Copy link
Member

Lyaaaaaaaaaaaaaaa commented May 8, 2024 •
edited
Loading

Describe the bug
The server crashes when using float16 without CUDA.

To Reproduce
Steps to reproduce the behavior:

  1. Run without CUDA
  2. Load an AI with float16
  3. Generate something
  4. See error

Expected behavior
It should fall back to float32 to avoid crash.

UserWarning: You are calling .generate() with the input_ids being on a device type different than your model's device. input_ids is on cpu, whereas the model is on cuda. You may experience unexpected behaviors or slower generation. Please make sure that you have put input_ids to the correct device by calling for example input_ids = input_ids.to('cuda') before running .generate().
warnings.warn(
"topk_cpu" not implemented for 'Half'
@Lyaaaaaaaaaaaaaaa Lyaaaaaaaaaaaaaaa added the bug Something isn't working label May 8, 2024
@Lyaaaaaaaaaaaaaaa Lyaaaaaaaaaaaaaaa self-assigned this May 8, 2024
Lyaaaaaaaaaaaaaaa added a commit that referenced this issue May 8, 2024
@Lyaaaaaaaaaaaaaaa
Added a fallback to change torch_dtype if cuda isn't available.
e41567f 
config.py:
- Added TORCH_DTYPE_SAFETY.

model.py:
- Updated _load_model to force (if config.TORCH_DTYPE_SAFETY is True)
torch_dtype to be set to float32 if cuda isn't available.
Because otherwise, it will lead to an error during generation.
See #31
@Lyaaaaaaaaaaaaaaa Lyaaaaaaaaaaaaaaa mentioned this issue May 10, 2024
Merged
@Lyaaaaaaaaaaaaaaa
Copy link
Member Author

Lyaaaaaaaaaaaaaaa commented May 10, 2024 •
edited
Loading

"Fixed" by commit 6b76d8d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Lyaaaaaaaaaaaaaaa Lyaaaaaaaaaaaaaaa

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Lyaaaaaaaaaaaaaaa