Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model request: Ovis1.6-Gemma2-9B-bnb-4bit #22

Open
Jonseed opened this issue Oct 9, 2024 · 3 comments
Open

Model request: Ovis1.6-Gemma2-9B-bnb-4bit #22

Jonseed opened this issue Oct 9, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@Jonseed
Copy link

Jonseed commented Oct 9, 2024

It would be great to add this 4-bit quantized version of Ovis 1.6, to run on lower memory: https://huggingface.co/ThetaCursed/Ovis1.6-Gemma2-9B-bnb-4bit

@Jonseed Jonseed changed the title Model request: ThetaCursed/Ovis1.6-Gemma2-9B-bnb-4bit Model request: Ovis1.6-Gemma2-9B-bnb-4bit Oct 9, 2024
@matatonic
Copy link
Owner

I tried this model, I had the same issue as with --load-in-4bit, it had a type conflict. you can try to load it yourself, without any extra arguments, it doesn't work. This is something that I think the model maker will need to fix, but if anyone knows a fix I would be happy to make the changes.

@Jonseed
Copy link
Author

Jonseed commented Oct 9, 2024

The model maker said "The issue arises during the image conversion process for the visual tokenizer. The preprocess_image function in the modeling_ovis.py script fails to properly convert the images to the required format or type for the visual tokenizer." They then said they got it to work. Maybe they would be willing to share how they fixed it.

@matatonic matatonic added the enhancement New feature or request label Oct 13, 2024
@Jonseed
Copy link
Author

Jonseed commented Nov 4, 2024

There are now official 4-bit versions available:

https://huggingface.co/AIDC-AI/Ovis1.6-Gemma2-9B-GPTQ-Int4

https://huggingface.co/AIDC-AI/Ovis1.6-Llama3.2-3B-GPTQ-Int4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants