lora : fix llama conversion script with model having ROPE_FREQS #9117
Conversation
github-actions
bot
added
testing
| if not self.is_lora: | ||
| self.gguf_writer.add_tensor(self.format_tensor_name(gguf.MODEL_TENSOR.ROPE_FREQS), np.array(rope_factors, dtype=np.float32)) |
There was a problem hiding this comment.
For Phi-3, vocab-only conversion is also affected by these rope_freqs tensors, because this is in set_gguf_parameters. (which makes vocab-only Phi-3-128k models produce invalid GGUF files (this is already a problem on master))
A more general solution to both LoRA and vocab-only conversions should be possible.
Maybe some kind of self.generate_extra_tensors() which would be called by self.prepare_tensors() before it calls self.get_tensors(). And LoraModel could simply override generate_extra_tensors() to a no-op (and vocab-only conversion does not call prepare_tensors). It can be done in a follow-up PR, though.
There was a problem hiding this comment.
OK I'll merge this now and will let you refactor this further in a follow-up PR.
Thank you for the help!
Resolve #9114