Model versus tokenizer mismatch

[petergreis]

Greetings

Attempting to load a fine tuned model into llama.cpp. As the error sources from my sft ChatMusician model I post this here. I went back to try this on a saved copy of the model.

Running this:
python3 convert.py /Users/petergreis/Dropbox/Leeds/Project/chatmusician_model_tokenizer

Yields this:

ValueError: Vocab size mismatch (model has 32000, but /Users/petergreis/Dropbox/Leeds/Project/chatmusician_model_tokenizer/tokenizer.model has 32001).

And in the model directory itself I see:

 % more added_tokens.json
{
  "<pad>": 32000
}

Which explains why the token count is off by one. Any idea how I can get the two to agree?

[hf-lin]

I find the same issue：ggerganov/llama.cpp#4045 (comment)