Provide .GGUF files?

[flatsiedatsie]

Would it be possible to provice a full range of GGUF files for these wicked models?

I'm tyring to convert the 3B myself, but running into issues.

[mayank31398]

@flatsiedatsie some folks in the community have already started working on this
relevant discussion: ggerganov/llama.cpp#7116

[YorkieDev]

@flatsiedatsie @mayank31398

I've made a couple of GGUF's here:

https://huggingface.co/YorkieOH10/granite-8b-code-instruct-Q8_0-GGUF
https://huggingface.co/YorkieOH10/granite-8b-code-instruct-Q4_K_M-GGUF
https://huggingface.co/YorkieOH10/granite-34b-code-instruct-Q8_0-GGUF

But waiting on support in llama.cpp before they can be used.

[mayank31398]

Thanks @YorkieDev
I am not very familiar with GGUF but whats the difference between Q8_0 and Q4_K_M ?
Also, is this the required PR to merge? ggerganov/llama.cpp#7116

[mayank31398]

Also, not sure if its hard, but would be awesome if you can create GGUFs of all the models

[YorkieDev]

@mayank31398 That's correct, once that PR's been merged GGUF's will work. I can get some more quants made up of them.

From my understanding, it's a scale of quality/size Q8 is heavier to run, but Q4 is lighter, and is what the majority of folks use.

For more info on GGUF models: https://huggingface.co/docs/hub/gguf

[mayank31398]

thanks for the explanation.

[flatsiedatsie]

Having a wider range of quants would rock.

For example, I would like to use Granite in a browser-based project. In that use case it's optimal when a file it below 2Gb in size, and the Q4 quant you provided is just above that threshold.

[mayank31398]

GGUFs are now available: https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330