support loading vocab from fast tokenizer config in convert.py

[strutive07]

Thank you for providing such a great repo.

The huggingface tokenizer works by calling AutoTokenizer.from_pretrained based on vocab.json and tokenizer.json. For llama2, it supports tokenizer.model for sentencepiece tokenizer, so llama.cpp supports loading vocab with tokenizer.model with sentencepiece tokenizer.

In multi lingual situation, there are cases of models that add additional vocab to tokenizer to train PLM further. In this case, we cannot support tokenizer.model, only AutoTokenizer.from_pretrained, which is based on vocab.json and tokenizer.json.

Therefore, there may be cases where there is no tokenizer.model file, so i propose to add a vocab class(in convert.py), 'HFVocab', to support this in this PR.

Related issue

• feature request - support huggingface tokenizer #1764

I tested this with the llama2 model kfkas/Llama-2-en-7b-Chat, where Korean vocab is added to the existing llama2 vocab. In my test, I tried to see if it would generate based on the added Korean vocab. You can see that the results are generated well.

Convert

python convert.py ../llama-2-ko-7b-chat --vocabtype hf

Prompt

./main -m ../llama-2-ko-7b-chat/ggml-model-f16.gguf \
    --prompt "아래는 작업을 설명하는 명령어입니다. 요청을 적절히 완료하는 응답을 작성하세요.\n\n### 명령어:\n삼원색에 대해 알려줘\n\n### 응답:"

To make it easier for maintainers to check their PRs, we've added a translation of the prompt and results below.

Below is a command that describes the task. Write a response that appropriately completes the request.\n\n### Command:\nTell me about the trichromatic colors\n\n### Response:

Original Result

아래는 작업을 설명하는 명령어입니다. 요청을 적절히 완료하는 응답을 작성하세요.\n\n### 명령어:\n삼원색에 대해 알려줘\n\n### 응답:현재 제공한 정보로는 삼원색을 정확하게 정의할 수 없습니다. 그러나 색상의 3가지 기본 요소는 빨간색, 파란색, 초록색이므로 이 세 가지 원소를 가지고 있는 색이 삼원색일 가능성이 높습니다.삼원색이 가장 일반적으로 사용되는 이유는 색이 더 복잡해질수록 다양한 색상이 혼합되어 더 많은 파장을 방출하기 때문입니다. 즉, 더 많은 빛의 스펙트럼이 섞이면 특정 색상에 대한 감각이 더 혼란스러워지고 색상을 정확하게 인식하는 것이 어려워집니다.예를 들어, 빨간색, 파란색 및 초록색을 포함하는 삼원색 계열인 오렌지, 블루 또는 그린의 조합으로 그린이나 오렌지처럼 복잡한 색상의 조합을 만들 수 있습니다. 이 경우 각각의 원소가 모두 혼합되어 새로운 색상이 만들어지기 때문에 더 많은 색상을 혼합하더라도 항상 특정 색상에 대한 감각을 유지할 수 있습니다.또한, 삼원색은 시각적 혼란이나 감각 상실 없이 다양한 색상을 혼합하여 다른 색상을 만들 때 중요한 역할을 할 수 있습니다. 예를 들어, 파란색을 혼합하면 보라색으로, 빨간색을 혼합하면 갈색으로, 초록색을 혼합하면 노랑색으로 변합니다. 이러한 색상은 모두 기본 삼원색을 기반으로 하고 있으므로 서로 다른 색상으로 인식할 수 있습니다.또한 삼원색은 많은 미술 작품과 예술가들이 사용하는 주요 색상의 조합이기 때문에 예술 분야에서 사용할 수 있는 유용한 색상 조합을 제공할 수 있다는 점에서도 중요합니다.

Result (translated for maintainers)

Below is a command that describes the task. Compose a response that appropriately completes the request.\n\n### Command:\nTell me about the trichromatic colors\n\n### Response:With the information you have provided, we cannot accurately define trichromatic colors. However, the three basic elements of color are red, blue, and green, so any color that has these three elements is likely a trichromatic color.The reason trichromatic colors are most commonly used is that as colors become more complex, they emit more wavelengths as different colors are mixed together. This means that when more spectrums of light are mixed together, the sensation of a particular color becomes more confusing and it becomes difficult to accurately perceive the color.For example, a combination of orange, blue, or green, which is a trichromatic color family that includes red, blue, and green, can create a combination of colors as complex as green or orange. In this case, you can always maintain a sense of a particular color, even as you mix more colors, because each element is mixed together to create a new color.Additionally, trichromats can play an important role in mixing different colors to create other colors without causing visual confusion or loss of sensation. For example, mixing blue makes purple, mixing red makes brown, and mixing green makes yellow. These colors are all based on the primary trichromatic colors, so they can be perceived as different colors.It's also important to note that the trichromatic colors can provide useful color combinations that can be used in the arts, as they are the primary color combinations used by many works of art and artists.

[TheBloke]

Excellent, thank you!

FYI, it is possible for models with extra vocab to support tokenizer.model - you can make your own new SPM vocab, like TigerResearch have done - they have a Llama 2 based tokenizer.model with 60515 tokens: https://huggingface.co/TigerResearch/tigerbot-70b-chat-v2/blob/main/tokenizer.model

(see how much bigger it is than the default Llama tokenizer.model - nearly double the size)

But that is an extra step required and not all model creators may do this, so I fully support convert.py being able to load tokenizer.json

I have recently encountered a few models that included tokenizer.json but no tokenizer.model. For those models I have to copy in the original tokenizer.model from the base model (Llama, Llama 2 or Mistral). That works fine as long as they've not added vocab, but as you say, some models are now adding many new tokens to the vocab. So I have been fearing that in future there will be a model released that adds new vocab, but does not update tokenizer.model with it, meaning I could not create a GGUF for it without making my own SPM tokenizer.model.

So I welcome this change, thank you - it will be great to have an option to use the HF tokenizer.json instead of tokenizer.model.

Perhaps you could add logic that looks for tokenizer.json if tokenizer.model is not found? So tokenizer.model is tried first, but if it cannot be found, tokenizer.json is then looked for?

[TheBloke]

EDIT: please ignore the below, I hadn't seen that this is already fixed by another PR: #3585

Also there is currently an issue with convert.py caused by the recent change to HF Transformers - added_tokens.json now includes the standard special tokens, like this: https://huggingface.co/jondurbin/airoboros-m-7b-3.1/blob/main/added_tokens.json :

{
  "</s>": 2,
  "<s>": 1,
  "<unk>": 0
}

The problem is that the convert.py code for checking added tokens does not like it when it finds token IDs that are < the vocab_size. It will just error out and not try to make the FP16. So I have to handle this in my own code, removing any tokens with an ID < vocab_size from added_tokens.json, and re-saving it before convert.py is called.

I know this is not directly related to your change, but while you are working on convert.py and vocab, would it be possible to also address this issue?

[strutive07]

Is it okay to understand that the problem is caused by duplicate vocab in the original vocab and added_tokens.json?
In BpeVocab, SentencePieceVocab, etc., vocab_size is simply the sum of the sizes of the two vocab list, but if i deduplicate the (vocab, vocab_id) pair and get the size, it will be solved, but am I understanding correctly?

If I understand correctly, I think it would be good to handle the issue in the next PR after this PR merge.
The solution seems to be simple, so I think I can handle it after the HFVocab implementation merge.

[strutive07]

Perhaps you could add logic that looks for tokenizer.json if tokenizer.model is not found? So tokenizer.model is tried first, but if it cannot be found, tokenizer.json is then looked for?

I tried converting the two models below with the same code back to back to see if it works as an inference. The models are successfully converted and generate well.

meta-llama/Llama-2-13b-chat-hf

python convert.py /some_dir/llama-2-13b-chat-hf


Loading model file /some_dir/llama-2-13b-chat-hf/model-00001-of-00003.safetensors
Loading model file /some_dir/llama-2-13b-chat-hf/model-00001-of-00003.safetensors
Loading model file /some_dir/llama-2-13b-chat-hf/model-00002-of-00003.safetensors
Loading model file /some_dir/llama-2-13b-chat-hf/model-00003-of-00003.safetensors
params = Params(n_vocab=32000, n_embd=5120, n_layer=40, n_ctx=4096, n_ff=13824, n_head=40, n_head_kv=40, f_norm_eps=1e-05, f_rope_freq_base=None, f_rope_scale=None, ftype=None, path_model=PosixPath('/some_dir/llama-2-13b-chat-hf'))
Loading vocab file '/some_dir/llama-2-13b-chat-hf/tokenizer.model', type 'spm'

kfkas/Llama-2-ko-7b-Chat

python convert.py /some_dir/llama-2-ko-7b-chat/


Loading model file /some_dir/llama-2-ko-7b-chat/model-00001-of-00002.safetensors
Loading model file /some_dir/llama-2-ko-7b-chat/model-00001-of-00002.safetensors
Loading model file /some_dir/llama-2-ko-7b-chat/model-00002-of-00002.safetensors
params = Params(n_vocab=46336, n_embd=4096, n_layer=32, n_ctx=2048, n_ff=11008, n_head=32, n_head_kv=32, f_norm_eps=1e-05, f_rope_freq_base=None, f_rope_scale=None, ftype=None, path_model=PosixPath('/some_dir/llama-2-ko-7b-chat'))
Loading vocab file '/some_dir/llama-2-ko-7b-chat', type 'hf'

[TheBloke]

Is it okay to understand that the problem is caused by duplicate vocab in the original vocab and added_tokens.json? In BpeVocab, SentencePieceVocab, etc., vocab_size is simply the sum of the sizes of the two vocab list, but if i deduplicate the (vocab, vocab_id) pair and get the size, it will be solved, but am I understanding correctly?

If I understand correctly, I think it would be good to handle the issue in the next PR after this PR merge. The solution seems to be simple, so I think I can handle it after the HFVocab implementation merge.

Actually please don't worry. I didn't realise there was already another PR to address this issue: #3585

[teleprint-me]

I'm working on an experimental solution to this problem because I keep running into it.

I'm confident there's a way to do this without creating dependencies.

We technically do not need to rely on huggingface and I can actually see reliance on it becoming an issue of its own.

I'm in the middle of creating some utilities to dump the necessary data to mapped data structures for reuse; think of it like a programmitic hexdump, but for models.

I already created one for safetensors. My next goal is to handle it for torch models. Then for huggingface models.

If my intuition is correct, then we shouldn't really need huggingface at all which would actually be a really good thing.

It would also be flexible enough to build on top of and extend as needed.

It would create a gateway towards unifying and streamlining all model conversions as well, which is my end goal.

[ggerganov]

@strutive07 Thank you for the detailed description and the contribution

Is there a way to make the transformers import executed only when parsing an HF vocab?
The goal is to avoid the dependency on this package in cases where we don't need it.

@teleprint-me Can't tell yet if your approach would be better. AFAICT you avoid the dependency, but there is a lot of custom format handling that needs to happen that would increase significantly the amount of python code. So not sure.

Curious what other people think - should we add transformers to requirements.txt?

[cebtenzzre]

Curious what other people think - should we add transformers to requirements.txt?

I am fine with importing transformers in this special case (vocabtype == "hf") - which is definitely possible. I don't want us to have to maintain too much extra code to duplicate the behavior of transformers in edge cases.

[teleprint-me]

@ggerganov

After spending the last few days on it, I agree. It's better to just include safetensors, transformers, and torch. convert.py handles a lot of the custom code and I now understand why it is the way it is. I was hoping there was a middle ground, but due to underlying complexity of torch, I readjusted my position. It would both reduce the amount of python code necessary and simplify the implementation as a result.

Doing so would allow a simpler implementation for a factory pattern to handle the variety of models and conversions required. I'm still researching and studying the source code.

[strutive07]

@ggerganov

Is there a way to make the transformers import executed only when parsing an HF vocab?
The goal is to avoid the dependency on this package in cases where we don't need it.

I move transformers import code into Class __init__ method.

Curious what other people think - should we add transformers to requirements.txt?

I don't think we need to add it to requirements.txt. I looked at other open source code that was similar to this situation. In huggingface/peft repo's bitsandbytes import case (import code) (dependency code), it doesn't add bitsandbytes to dependency code and only checks it when it's actually needed, providing import errors and installation code.

We don't need to add it to our dependencies in a similar way, we just need to try to import it when it's needed, and if it's not, provide an import error and install code.

I wrote some code based on the above thoughts and committed to this PR. (Code)

[ggerganov]

Thank you, I think this OK to merge.
Will wait for a bit to see if there are any concerns from others and we can merge

[teleprint-me]

If it's alright, I'd like to play devils advocate.

Why omit it from the dependencies, create an exception that requires the dependency, only to have the user install it manually in the end?

In most cases, if you're using the python scripts, you'll need the deps. So, why not include them to begin with? It feels like a "false start" simply because I'll need to install it anyways.

The other conversion scripts are in the same boat as well.

I'm simply stating this because I think it's worth genuine consideration.

[cebtenzzre]

Why omit it from the dependencies, create an exception that requires the dependency, only to have the user install it manually in the end?

I believe most users will not run into this issue. I have never personally come across a SPM-based model from HF without a tokenizer.model. Putting it in requirements.txt implies that it is a required dependency, not an optional dependency.

[teleprint-me]

Why omit it from the dependencies, create an exception that requires the dependency, only to have the user install it manually in the end?

I have never personally come across a SPM-based model from HF without a tokenizer.model.

Why? What's the difference? I thought the point was to support HF tokenizers which are missing the model and have the vocab instead?

[cebtenzzre]

Why? What's the difference? I thought the point was to support HF tokenizers which are missing the model and have the vocab instead?

Yes, that's what this PR is for, I'm just saying that it solves a problem that I believe very few people have. If the tokenizer.model is missing, it was manually removed and replaced with a vocab in the tokenizer.json, and that is not the only way to do it: #3633 (comment)

[teleprint-me]

@cebtenzzre

So I have been fearing that in future there will be a model released that adds new vocab, but does not update tokenizer.model with it, meaning I could not create a GGUF for it without making my own SPM tokenizer.model.

So I welcome this change, thank you - it will be great to have an option to use the HF tokenizer.json instead of tokenizer.model.

You did notice that I liked the comment, right? Also, I don't see the relevance or how it answers my question(s).

If the tokenizer.model is missing, it was manually removed and replaced with a vocab in the tokenizer.json, and that is not the only way to do it: comment#3633

It may not necessarily have been removed and may have not previously existed. Regardless, something is still required to fill the void.

I find these counter-points to be counterproductive and I fail to see how they should be dissuading.

None of this changes the fact that the conversion scripts are becoming increasingly reliant on these 3rd party libraries simply because shimming/replicating them is proving to be cumbersome; The simpler approach is to include them.

[TheBloke]

@teleprint-me this is quite common. Model creators often bump the tensor sizes without an according vocab increase. In this case it was likely done to ensure the tensors are a multiple of 256, enabling tensor parellelism and GGUF k-quants.

This is no issue with Transformers, but has been a problem in llama.cpp since the beginning. In the main branch, we have to deal with this externally, by adding dummy tokens to added_tokens.json.

In this PR, a pad vocab feature first implemented by Kerfuffle has been added to allow convert.py to fix it for you.

So, just add --padvocab to your converrt.py command line and it will be fixed.

I believe there's discussions going elsewhere in this Gitub regarding a refactor of vocab handling which might eventually make this vocab padding unnecessary.

[TheBloke]

Hi @strutive07

I've had a few models recently that lacked tokenizer.model and so I did with this PR. I've had reports on some that they are not capable of encoding \n, and instead they write <0x0A.

There's an increasing number of models not releasing tokenizer.model so this PR is proving extremely useful, but it seems that in some situations a fix to the output is needed and I'm not sure if it's the model's fault, or a bug in this PR?

Here is an example model commit which has this issue: https://huggingface.co/argilla/notus-7b-v1/tree/83af961763096872675626d26ae5b735c61caa49

I've linked to a revision as they've since uploaded tokenizer.model, changed tokenizer_config.json, and add added_tokens.json, so it can now be converted with main convert.py.

The linked revision is how I first converted the model, and is the one that outputs <0x0A> in place of \n.

Here is a demonstration of the problem, using the above model revision:

Convert:

 ᐅ python3 ./convert-fast.py /workspace/process/test-notus-7b --outtype f16 --outfile /workspace/process/notus-7b-v1.issue.fp16.gguf
Loading model file /workspace/process/test-notus-7b/model-00001-of-00003.safetensors
Loading model file /workspace/process/test-notus-7b/model-00001-of-00003.safetensors
Loading model file /workspace/process/test-notus-7b/model-00002-of-00003.safetensors
Loading model file /workspace/process/test-notus-7b/model-00003-of-00003.safetensors
params = Params(n_vocab=32000, n_embd=4096, n_layer=32, n_ctx=32768, n_ff=14336, n_head=32, n_head_kv=8, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=10000.0, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=<GGMLFileType.MostlyF16: 1>, path_model=PosixPath('/workspace/process/test-notus-7b'))
Vocab info: <VocabLoader with 32000 base tokens and 0 added tokens>
Special vocab info: <SpecialVocab with 58980 merges, special tokens {'bos': 1, 'eos': 2, 'unk': 0, 'pad': 2}, add special tokens unset>

... output trimmed ...

Writing /workspace/process/notus-7b-v1.issue.fp16.gguf, format 1
gguf: This GGUF file is for Little Endian only
gguf: Adding 58980 merge(s).
gguf: Setting special token type bos to 1
gguf: Setting special token type eos to 2
gguf: Setting special token type unk to 0
gguf: Setting special token type pad to 2
gguf: Setting chat_template to {% for message in messages %}
{% if message['role'] == 'user' %}
{{ '<|user|>
' + message['content'] + eos_token }}
{% elif message['role'] == 'system' %}
{{ '<|system|>
' + message['content'] + eos_token }}
{% elif message['role'] == 'assistant' %}
{{ '<|assistant|>
'  + message['content'] + eos_token }}
{% endif %}
{% if loop.last and add_generation_prompt %}
{{ '<|assistant|>' }}
{% endif %}
{% endfor %}
...
[291/291] Writing tensor output_norm.weight                     | size   4096           | type F32  | T+  19
Wrote /workspace/process/notus-7b-v1.issue.fp16.gguf

Test inference using FP16

ᐅ CUDA_VISIBLE_DEVICES=1 ./main -m /workspace/process/notus-7b-v1.issue.fp16.gguf -t 1  -ngl 100  -p "<|system|>
You are a story writing assistant</s>
<|user|>
Write a story about llamas</s>
<|assistant|>"

...
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: PAD token        = 2 '</s>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.11 MiB
...
<|system|><0x0A>You are a story writing assistant<0x0A><|user|><0x0A>Write a story about llamas<0x0A><|assistant|><0x0A>Deep in the Andean mountains, there lived a herd of llamas. They roamed free on the grassy meadows and basked in the warm sun. The llamas were known for their gentle nature, sturdy build, and long memories. They had learned to navigate through the treacherous terrain, avoiding steep cliffs and rocky paths. Their thick woolly coats kept them warm during chilly nights, and their keen senses helped them locate food and water sources.<0x0A><0x0A>One .....

Is it an issue with their original config, or is it this PR?

[strutive07]

@TheBloke I compared the master branch and vocab using tokenizer.model and they are exactly the same. It doesn't seem to be an issue with the new convert.py.

from convert import VocabLoader, Params, load_some_model, check_vocab_size
from convert_master_branch import load_vocab as load_vocab_master_branch

def is_same_vocab(v1, v2):
    v1_set = set()
    v2_set = set()

    for text, score, toktype in v1.all_tokens():
        v1_set.add((text, score, toktype))
        
    for text, score, toktype in v2.all_tokens():
        v2_set.add((text, score, toktype))

    if v1_set != v2_set:
        v1_dict = {}
        
        for text, score, toktype in v1.all_tokens():
            v1_dict[text] = (score, toktype)
            
        for text, score, toktype in v2.all_tokens():
            if text not in v1_dict:
                print(text.decode('utf-8'), 'not in ', text, score, toktype)
            else:
                if (score, toktype) != v1_dict[text]:
                    print(text.decode('utf-8'), 'diff', (score, toktype), v1_dict[text])
    return v1_set == v2_set


def check(model_path):
    params = Params.load(load_some_model(model_path))
    vocab_new = VocabLoader(params, model_path)
    vocab_orig = load_vocab_master_branch(model_path, "spm")
    print(model_path)
    print(is_same_vocab(vocab_new, vocab_orig))

from pathlib import Path
model_path = Path('notus-7b-v1')
check(model_path)

Loading model file notus-7b-v1/model-00001-of-00003.safetensors
Loading model file notus-7b-v1/model-00001-of-00003.safetensors
Loading model file notus-7b-v1/model-00002-of-00003.safetensors
Loading model file notus-7b-v1/model-00003-of-00003.safetensors

32000 32000
Loading vocab file 'notus-7b-v1/tokenizer.model', type 'spm'
notus-7b-v1
True

[TheBloke]

Hi @strutive07 thanks very much for checking. It's great to know that tokenizer.model is the same.

But there is still an issue when tokenizer.json is used, and this is where it would be good to know whether there is a PR bug, or some issue in the base model.

Here is how this can be confirmed:

1. Download the current repo to local

pip3 install --upgrade 'huggingface-hub>=0.18'     # if not installed
huggingface-cli download argilla/notus-7b-v1 --local-dir test-notus  --local-dir-use-symlinks False

2. Delete tokenizer.model to force tokenizer.json to be used

rm test-notus/tokenizer.model

3. Run convert.py from this PR

python3 ./convert-fast.py /workspace/test-notus --outtype f16 --outfile /workspace/test-notus.fp16.gguf

4. Test inference and note that \n is not present in the output:

CUDA_VISIBLE_DEVICES=1 ./main -m /workspace/test-notus.fp16.gguf -t 1  -ngl 100  -p "<|system|>
You are a story writing assistant</s>
<|user|>
Write a story about llamas</s>
<|assistant|>"

...

 <|system|><0x0A>You are a story writing assistant<0x0A><|user|><0x0A>Write a story about llamas<0x0A><|assistant|><0x0A>Deep in the Andes mountains of South America, lived a herd of llamas. They were known for their long ears, sturdy legs, and fluffy coats that kept them warm during the harsh winters. The herd consisted of mothers, fathers, babies, and even a few mischievous young ones who loved to play pranks on each other.<0x0A><0x0A>One day, while grazing in a ....

So it would be good to know whether this repo has some problem with their tokenizer.json that is not present in the tokenizer.model? Or if the PR is causing the problem?

Because there are an increasing number of models not shipping tokenizer.model, so the issue could occur again and if it does, I currently don't know where it comes from.

Thanks again

[TheBloke]

Hi @strutive07

Here's another example: model https://huggingface.co/rwitz/go-bruins-v2 does not include tokenizer.model and when I make it with this PR, it has the <0x0A issue.

So speaking more generally: I think all Mistral models have this issue when converted from tokenizer.json instead of tokenizer.model.

In most cases I can manually resolve this by copying in the tokenizer.model from Mistral.

However:

1. It means that my code will by default produce bad GGUFs for models that don't include tokenizer.model, unless I put in a special check for "If model is Mistral 7B and tokenizer.model is missing, copy in the default tokenizer.model"
2. But then a few models have customised tokenizers, with increased numbers of tokens. And they don't include tokenizer.model because they have expanded tokenizer.json. So if I copied in default tokenizer.model for those models, that would be wrong
3. I can also check the vocab size and not copy it in if vocab_size is much bigger than 32000
4. But then it also raises the question: for any Mistral models with customised tokenizer and no customised tokenizer.model (like some Chinese models do) will they also have this <0x0A> issue? I think they will.

Note that using HF AutoTokenizer with this model shows no issue - newline decodes to \n OK.

Could it be some difference between how Mistral encodes newline vs Llama?

But still, I think the PR needs to be able to decode correctly otherwise it can't produce valid Mistral models from tokenizer.json.

[TheBloke]

OK this is interesting. I just learned that ExLlamav2's tokenizer.json conversion also has the <0x0A> issue when an ExLlama model is made from Mistral tokenizer.json.

So this does sound like it's not the fault of this PR.

But I don't understand because when I do it with AutoTokenizer, it's fine:

# This folder has no `tokenizer.model` - I deleted it
In [4]: tokenizer = AutoTokenizer.from_pretrained("/workspace/test-notus", use_fast=True, legacy=False, trust_remote_code=True)

In [7]: tokenizer.decode(tokenizer("""Hello how are you?
   ...: I am fine thanks
   ...: how are you?""").input_ids)
Out[7]: '<s> Hello how are you?\nI am fine thanks\nhow are you?'

In [8]: tokenizer.decode(13)
Out[6]: '\n'

[ArthurZucker]

Mmm the <0x0A> is a bytefallback of the \n character, which is correctly decoded by sentencepiece and the AutoTokenizer as \n, which means there might be an issue

[ggerganov]

Do you guys think the \n / <0x0A> could be addressed from master after we merge the PR?
I think it would be better to merge it in order to avoid stalling for too long and things becoming obsolete. Hopefully the newline issue could be resolved easily and might even get more attention from devs if it is present from the main line?

[strutive07]

@ggerganov cc. @TheBloke
I think it would be better to do the '\n / <0x0A> new line issue' in a new PR after this PR merge. It would be great to talk about it with others, but this PR has gotten too long for that. I still haven't figured out why it's behaving this way, even though the vocab is the same. I will be checking this issue most urgently.

[TheBloke]

Sounds good to me. It seems that all models that convert.py can currently handle will be handled identically with this PR. The issue only occurs with models without tokenizer.model, which current convert.py can't handle anyway

So there is no regression, just an issue affecting some of the extra models that convert.py will now be able to handle.

And from my perspective it'd be great for this to be merged so I get tokenizer.json support and --padvocab in main. Even if Mistral models have issues, there are still plenty of Llama-based models that lack tokenizer.model, and don't have the \n issue.

Thanks for all your long work on this @strutive07 !

[ggerganov]

Alright sounds good. Waiting for @cebtenzzre to approve and we should merge

[teleprint-me]

I made my own version of the convert.py and haven't had any issues regarding whether the tokenizer.model is available or not. It smoothly handles most known cases with the exception that I omitted GPT2 support from it. Once this PR is merged, I can take a look at it and see what's going on in there once I have some time.

[ggerganov]

The ggml CI is now failing with the following error:

+ python3 ../convert.py ../models-mnt/open-llama/3B-v2
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Loading model file ../models-mnt/open-llama/3B-v2/pytorch_model.bin
params = Params(n_vocab=32000, n_embd=3200, n_layer=26, n_ctx=2048, n_ff=8640, n_head=32, n_head_kv=32, n_experts=None, n_experts_used=None, f_norm_eps=1e-06, rope_scaling_type=None, f_rope_freq_base=None, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=None, path_model=PosixPath('../models-mnt/open-llama/3B-v2'))
Traceback (most recent call last):
  File "/home/ggml/work/llama.cpp/build-ci-release/../convert.py", line 1279, in <module>
    main()
  File "/home/ggml/work/llama.cpp/build-ci-release/../convert.py", line 1255, in main
    vocab = VocabLoader(params, vocab_dir)
  File "/home/ggml/work/llama.cpp/build-ci-release/../convert.py", line 342, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(str(fname_tokenizer), trust_remote_code=True)
  File "/home/ggml/.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 787, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/ggml/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2028, in from_pretrained
    return cls._from_pretrained(
  File "/home/ggml/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2260, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/ggml/.local/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 124, in __init__
    super().__init__(
  File "/home/ggml/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 114, in __init__
    fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
  File "/home/ggml/.local/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py", line 1336, in convert_slow_tokenizer
    return converter_class(transformer_tokenizer).converted()
  File "/home/ggml/.local/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py", line 459, in __init__
    requires_backends(self, "protobuf")
  File "/home/ggml/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1276, in requires_backends
    raise ImportError("".join(failed))
ImportError: 
LlamaConverter requires the protobuf library but it was not found in your environment. Checkout the instructions on the
installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones
that match your environment. Please note that you may need to restart your runtime after installation.

+ cur=1
+ echo 1
+ set +x

https://github.com/ggml-org/ci/blob/results/llama.cpp/87/3637afc7924f435ac44c067630a28e82eefa7b/ggml-3-arm64-cpu/stdall

Somehow I missed that this change also introduces dependency on protobuf.

[Ivanmatthew]

Can you help me understand why you removed *.safetensors?

Thanks!

[teleprint-me]

The model files are automatically discovered. The model file names are arbitrary, some are split and others are not. The model files are dynamically "lazy" loaded to save memory as a result.

https://github.com/ggerganov/llama.cpp/blob/master/convert.py#L1348

The vocab can be set manually though.

https://github.com/ggerganov/llama.cpp/blob/master/convert.py#L1461

Not sure if that answers your question.