Command-R GGUF conversion no longer working

[candre23]

As recently as a few days ago, Command-R (and presumably R+) could be converted with convert-hf-to-gguf.py. I double checked and conversion completes successfully in b2751. However, with the recent changes to accommodate Llama3, Command-R compatibility has been broken. Trying to convert today with b2777 I get

raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")
NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

I know that L3 required a new tokenizer provided by meta to facilitate proper conversion. Do we require something new from cohere, or is this something that can be fixed internally?

[drummerv]

I had the same issue and the same error! Had to rollback to before Llama 3 to get it to work.

[dranger003]

This patch appears to get it working again.

diff --git a/convert-hf-to-gguf.py b/convert-hf-to-gguf.py
index 2f146d73..e8cc040f 100755
--- a/convert-hf-to-gguf.py
+++ b/convert-hf-to-gguf.py
@@ -306,6 +306,8 @@ class Model(ABC):
         if chkhsh == "3ce83efda5659b07b1ad37ca97ca5797ea4285d9b9ab0dc679e4a720c9da7454":
             # ref: https://huggingface.co/openai-community/gpt2
             res = "gpt-2"
+        if chkhsh == "9c2227e4dd922002fb81bde4fc02b0483ca4f12911410dee2255e4987644e3f8":
+            res = "gpt-2"

         if res is None:
             print("\n")

[dranger003]

There seems to be an issue using convert-hf-to-gguf-update.py and the downloaded tokenizer.json file does not contain JSON but rather contains this:

version https://git-lfs.github.com/spec/v1
oid sha256:0af6e6fe50ce1bb5611b103482de6bac000c82e06898138d57f35af121aec772
size 12777406

This is after adding this to convert-hf-to-gguf-update.py:

{ "name": "command-r",      "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/CohereForAI/c4ai-command-r-v01", },

[candre23]

I downloaded the model manually earlier today and the tokenizer.json is definitely a real tokenizer. The file size mentioned in the placeholder that you got matches the actual size of what I got from cohere's HF repo (~12.1mb).

[sealad886]

There seems to be an issue using convert-hf-to-gguf-update.py and the downloaded tokenizer.json file does not contain JSON but rather contains this:

version https://git-lfs.github.com/spec/v1
oid sha256:0af6e6fe50ce1bb5611b103482de6bac000c82e06898138d57f35af121aec772
size 12777406

This is after adding this to convert-hf-to-gguf-update.py:

{ "name": "command-r",      "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/CohereForAI/c4ai-command-r-v01", },

This is caused by Git LFS (for more details). To fix this, download the tokenizer.json file directly from Huggingface and replace the file that was previously downloaded:

tokenizer.json for command-r
--or--
tokenizer.json for command-r-plus

[sealad886]

As @candre23 noted, this also affects command-r-plus...and every model of a specific type (BPE pre-tokenizer? I'm not super familiar with that).
It's a non-trivial but straightforward way to add back model types to convert-hf-to-gguf.py. Follow the instructions in this pull request. Admittedly, I got a bit lost trying to figure out what regex to use for the command-r-plus pre-tokenizer type (it appears to me that Cohere have made this one unified type?).
I also might have royallly screwed up my source files accidentally. I was too tired when I started looking at this, and by the end couldn't put one thought in front of another...

[sealad886]

As @candre23 noted, this also affects command-r-plus...and every model of a specific type (BPE pre-tokenizer? I'm not super familiar with that). It's a non-trivial but straightforward way to add back model types to convert-hf-to-gguf.py. Follow the instructions in this pull request. Admittedly, I got a bit lost trying to figure out what regex to use for the command-r-plus pre-tokenizer type (it appears to me that Cohere have made this one unified type?). I also might have royallly screwed up my source files accidentally. I was too tired when I started looking at this, and by the end couldn't put one thought in front of another...

Got this working myself about the same time I saw your PR, @dranger003. It does appear this is fixed by #7033 so let's get that merged asap. :)

[oldgithubman]

As @candre23 noted, this also affects command-r-plus...and every model of a specific type (BPE pre-tokenizer? I'm not super familiar with that). It's a non-trivial but straightforward way to add back model types to convert-hf-to-gguf.py. Follow the instructions in this pull request. Admittedly, I got a bit lost trying to figure out what regex to use for the command-r-plus pre-tokenizer type (it appears to me that Cohere have made this one unified type?). I also might have royallly screwed up my source files accidentally. I was too tired when I started looking at this, and by the end couldn't put one thought in front of another...

IMHO, this is all unreasonably complicated, yes. Plus the classic lack of clear documentation. I'm trying to wrap my head around all this and I get to this comment: "# - Copy-paste the generated get_vocab_base_pre() function into convert-hf-to-gguf.py"
Excuse me? Where is "the generated get_vocab_base_pre() function" we're meant to copy?