How to export quantized model after use QuaRot+GPTQ and infer with vllm

[hensiesp32]

Hi, I was trying to export an model with QuaRot+GPTQ and infer with vllm, however, there are some problems with vllm's infer.
KeyError: 'layers.0.mlp.down_proj.buf_invperm
the config is as below:

• step_1_quarot.yaml

base:
    seed: &seed 0
model:
    type: Llama
    path: /mnt/public/open_source_model/Llama-2-7b-hf
    tokenizer_mode: slow
    torch_dtype: auto
quant:
    method: Quarot
    weight:
        bit: 8
        symmetric: False
        granularity: per_channel
        group_size: -1
        calib_algo: minmax
    act:
        bit: 8
        symmetric: False
        granularity: per_token
    special:
        rotate_mode: hadamard
        fp32_had: True
        online_rotate: True
save:
    # Save the Quarot-transformed model.
    save_trans: True
    save_fake: False
    save_path: /mnt/public/daixin/Llama-2-7b-quarot-w8a8

• step_2_gptq.yaml

    seed: &seed 0
model:
    type: Llama
    # Load Quarot-transformed model
    path: /mnt/public/daixin/Llama-2-7b-quarot-w8a8/transformed_model
    torch_dtype: auto
calib:
    name: pileval
    download: False
    path: /mnt/public/lingo-engine/data/pileval_dataset/
    n_samples: 256
    bs: 1
    seq_len: 2048
    preproc: general
    seed: *seed
quant:
    method: GPTQ
    weight:
        bit: 8
        symmetric: False
        granularity: per_channel
        group_size: -1
        calib_algo: mse
    act:
        bit: 8
        symmetric: False
        granularity: per_token
        calib_algo: minmax
    special:
        actorder: True
        static_groups: True
        percdamp: 0.01
        blocksize: 128
        true_sequential: True
        online_rotate: True
        fp32_had: True
    quant_out: True
save:
    save_trans: True
    save_fake: False
    save_path: /mnt/public/daixin/Llama-2-7b-quarot+gptq-w8a8

I wonder whether LLMC can export Quarot quantized model that supports vllm inference.

[gushiqiao]

In step_2_gptq.yaml, save_vllm:True should be specified instead of save_trans:True

[gushiqiao]

llmc/configs/quantization/backend/vllm/w8a8_combin/step_2_gptq.yml

Line 52 in f60ddeb

save_vllm: True

[hensiesp32]

llmc/configs/quantization/backend/vllm/w8a8_combin/step_2_gptq.yml

thanks a lot ! I will try. By the way, can LLMC's Quarot supports Qwen2 that attention architecture is GQA.

[gushiqiao]

Quarot supports Qwen2 but does not support online_rotation. This limitation may be related to the shape of the weights, as the online Hadamard rotation imposes strict requirements on the shape.