cfg-cache not working

[Kaszebe]

I am using this quant: https://huggingface.co/BigHuggyD/anthracite-org_magnum-v2-123b_exl2_8.0bpw_h8

Mr. Ooobabooga said that enabling this and setting it to 1.5 can make the model smarter. This is according to a paper he was referencing in this thread: https://www.reddit.com/r/Oobabooga/comments/1cf9bso/what_does_guidance_scale_parameter_do/

I loaded up the above quant and checked the cfg-cache box. The model loaded just fine. I then went to the parameters tab and set the guidance_scale to 1.5. I left the 'negative prompt' box empty.

I then went to chat-instruct and asked the model a question. It appeared for a second that it was processing the prompt and then it did not return a single thing.

I am able to get the model working successfully and chatting with me if I uncheck the cfg-cache box when loading the quant

Am I doing something wrong here?

[jepjoo]

I loaded up the above quant and checked the cfg-cache box. The model loaded just fine. I then went to the parameters tab and set the guidance_scale to 1.5. I left the 'negative prompt' box empty.

I then went to chat-instruct and asked the model a question. It appeared for a second that it was processing the prompt and then it did not return a single thing.

On my end CFG does work when doing those steps you described.

Memory requirements increase with CFG and you are running a massive model, so I'm wondering if you might be hitting a memory limit.

[Kaszebe]

The quant is 100GB and I have 136GB of VRAM. Is that not enough? I had to lower context to 5096 to get it to load with CFG turned on.

[jepjoo]

I don't know if that's enough or not, but you could try with a small model like Llama 3 8B or something just to see if CFG works on your end at all.