Bugfix: Download index file for cases with multiple safetensors

[ceshine]

Please excuse me for presumptuously calling this patch a "bugfix." I am unsure if there is another method to correctly export fine-tuned LoRA weights for Qwen-2 VL models into full FP16 model weights that would be loadable by vLLM. However, this patch resolves the issue for me.

In summary, the save_pretrained_merged method of Qwen-2 model objects uses the merge_and_overwrite_lora function from this repository. This function downloads the safetensors files from the original FP16 model repository, applies updates to the weights using the provided LoRA weights, then writes the modified weights back into the downloaded safetensors files, if I have understood it correctly. However, the model index file model.safetensors.index is not being downloaded, and consequently, vLLM refuses to load the model due to the absence of this index file (I suspect the huggingface transformers library behaves similarly). The proposed solution is to download the index file alongside the safetensors files.

As I am not deeply familiar with this codebase, I apologize in advance if my understanding is incomplete. I hope this pull request aids someone encountering the same issue (vLLM refusing to load the merged model weights). Any feedback would be greatly appreciated.

Demo

Note: I manually removed the quantization_config section from the configuration file to enable vLLM to load the model properly.

[danielhanchen]

Extreme apologies on the delay - I'll definitely review the PR tomorrow - again sorry!

[danielhanchen]

@Erland366 Could you take a look at this PR thanks :)

[Erland366]

I still got an error of shape error. Which as you said, removing the quantization_config helps. We might need to remove that too before saving it

[ceshine]

@Erland366

Thanks for the feedback. I was unsure whether it was correct to remove that field, or the best way to do so in the code.

I added a commit (09fbe3f) to patch the exported config file if necessary, which seems to be the most non-intrusive way (i.e., not modifying existing code) to perform this removal. I ran a smoke test with the latest changes, and vllm can now load the model without manual intervention. Please take a look. Thank you.

[Erland366]

Can you test when saving LoRA, did you also save the config.json? (I worried that now we will save both config.json and adapter_config.json which will raise an error later on)

Then after merge from upstream/main. It good to go

I am sorry, usually I would do it myself but my CPU is so weak, merging always crash my PC and it'll took me at least 40 minutes for merging 2B models. So it's really hard to test :/

[ceshine]

@Erland366 No problem. The patch only affects the merge_and_overwrite_lora function, so it's unlikely to impact regular LoRA saving. Although the GitHub diff interface may suggest that the upload_items function is involved, it is actually unrelated.

It's reasonable to be cautious, though. I will provide the smoke test results and merge the main branch soon.

[ceshine]

@Erland366

Please see the screenshots below for the testing results. I have also merged the upstream changes into this branch.

[Erland366]

I think now all good @danielhanchen