Llava / Prismatic with LoRA

[gorjanradevski]

Great work! One quick question: in the paper you've reproduced the results from Llava. Additionally, for the Prismatic models experiments, you are fine-tuning the whole LM. I'm wondering did you try using LoRA in the LM for the Llava experiments as well as for the Prismatic models experiments? More specifically, why did you opt for the fine-tuning the entire LM instead of training LoRA modules?

Additionally, have you stored any training logs; e.g., weights & biases?

Thanks!

[NMS05]

Hi @gorjanradevski,

The weights and training logs are released by the authors in huggingface (https://huggingface.co/TRI-ML/prismatic-vlms/tree/main)

As for LoRA, I tried a simple implementation with SigLIP + Phi-2 + LoRA here (https://github.com/NMS05/Prismatic-SigLIP-Phi2-LoRA-VLM).

Thanks,

[zeyuanyin]

Hi, @NMS05. Thanks for your great implementation, which helps me a lot.

[siddk]

Awesome implementation @NMS05! Would love to add first-class LoRA support to this repo - would you be willing to contribute a PR?

[zeyuanyin]

Hi @NMS05, I would like to know if you have evaluated the finetuned model. I download and evaluate my_phi2_siglip_model via https://github.com/TRI-ML/vlm-evaluation. But the results are pretty bad. Or is there something wrong in eval repo? @siddk. Could you please double-check https://github.com/NMS05/Prismatic-SigLIP-Phi2-LoRA-VLM#model-weights and share your evaluation results? Thanks a lot.

	vqa-v2	gqa	vizwiz	text-vqa-ocr	text-vqa
phi2_siglip_lora (slim eval set)	6.950	3.520	0.290	0.283	0.068

BTW, it seems the lora results are worse than full fine-tune results on phi-2.

You can find my code at https://github.com/zeyuanyin/prismatic-vlms-phi-dev. The current progress is that I changed the LLM from Phi-1.5 to Phi-2, and the fine-tuned VLM model can generate the correct form answer without unrelated long sentences. But the evaluation results are pretty low, especially on text-vqa.

	vqa-v2	gqa	vizwiz	text-vqa-ocr	text-vqa
phi-v2-2.7b (slim eval set)	51.62	42.48	16.76	0.331	0.117

Originally posted by @zeyuanyin in #6 (comment)

[siddk]

Hmm so the evaluation code was able to fully reproduce the paper results for all existing models (as of last week); not sure if there's something different that needs to be changed for Phi-2 models (e.g., tokenization/generation parameters).

I'll try to take a look when I can (or might just try training a Phi-2 model). But would be great if @NMS05 could weigh in on their results!

[NMS05]

Hi @zeyuanyin and @siddk ,

Resolved the previous issue. The problems were 1) LoRA had fewer params and 2) full-finetune is not optimal.

Here is the latest model with improved design choices NMS05/DinoV2-SigLIP-Phi3-LoRA-VLM.

Evaluation using prismatic-eval. Results (Acc) for SLIM splits are as,

• VQAv2 - 63.3 %
• POPE final - 86.3%
• AI2D final - 58.9%

Thanks,