Deleting quantization_config broken

[psinger]

System Info

Version 4.47.0

Reproduction

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("h2oai/h2o-danube3-500m-chat", load_in_4bit=True)
del model.config.quantization_config
model.config

TypeError: Object of type dtype is not JSON serializable

Expected behavior

I am unable to delete the quantization_config from an existing model. Whenever I do it, it just completely breaks the whole config.

I also tried setting is_quantized=False but it does not change anything.

Is there another way of achieving this?

I am aware that there is a .dequantize function, but in this case Im changing dtypes on my own and want to exclude that quantization_config particularly when saving the model.

[Rocketknight1]

I think this might be a question for the forums/Discord, but pinging @SunMarc @MekkCyber just in case

[MekkCyber]

Hey @psinger, what's the reason you want to delete the quantization_config after the model is loaded ?

[psinger]

Because I am manually dequantizing and am not relying on the HF functionality for it. And then before pushing to hub, I want to remove the quantization_config from the config.

[MekkCyber]

Thanks for the clarification @psinger, I think you can make it work as follows :

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "h2oai/h2o-danube3-500m-chat", load_in_4bit=True
)

config_dict = model.config.to_dict()
if "quantization_config" in config_dict:
    del config_dict["quantization_config"]

from transformers import PretrainedConfig

model.config = PretrainedConfig.from_dict(config_dict)
print(model.config)

[SunMarc]

Does it fix the issue @psinger ?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.