HQQ vs exl2 Performance

[brodieferguson]

Just saw that HQQ support was added.
Really interesting idea + perplexity numbers for the 4/2 mix.
https://huggingface.co/mobiuslabsgmbh/Mixtral-8x7B-Instruct-v0.1-hf-attn-4bit-moe-2bit-HQQ

My system:
RTX 4090 + 13600k + 64GB DDR4 + WSL2

Testing Mixtral:

• 30-50 tokens/s on exl2 3.0bpw
• 3-5 tokens/s on 4/2 HQQ mix across the 3 loaders (ATEN & Pytorch / Compile)