Add phi-2 tokenizer

[BramVanroy]

This snippet yields an error:

python -c "
from huggingface_hub import snapshot_download;
snapshot_download(repo_id='microsoft/phi-2', local_dir='phi-2', local_dir_use_symlinks=False)
"
python convert-hf-to-gguf.py phi-2/ --outtype f16

Traceback (most recent call last):
  File "/home/local/vanroy/llama.cpp/convert-hf-to-gguf.py", line 3001, in <module>
    main()
  File "/home/local/vanroy/llama.cpp/convert-hf-to-gguf.py", line 2988, in main
    model_instance.set_vocab()
  File "/home/local/vanroy/llama.cpp/convert-hf-to-gguf.py", line 75, in set_vocab
    self._set_vocab_gpt2()
  File "/home/local/vanroy/llama.cpp/convert-hf-to-gguf.py", line 331, in _set_vocab_gpt2
    tokens, toktypes, tokpre = self.get_vocab_base()
  File "/home/local/vanroy/llama.cpp/convert-hf-to-gguf.py", line 242, in get_vocab_base
    tokpre = self.get_vocab_base_pre(tokenizer)
  File "/home/local/vanroy/llama.cpp/convert-hf-to-gguf.py", line 323, in get_vocab_base_pre
    raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")
NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

The proposed changes add support for phi-2, which uses CodeGenTokenizer, a BPE tokenizer.

closes #7022

[linpan]

raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")

NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

[ggerganov]

This new pre-tokenizer has to be handled in llama.cpp:

llama.cpp/llama.cpp

Lines 4414 to 4475 in e18bc6a

	// for now, only BPE models have pre-tokenizers
	if (vocab.type == LLAMA_VOCAB_TYPE_BPE) {
	if (tokenizer_pre.empty()) {
	LLAMA_LOG_WARN("%s: missing pre-tokenizer type, using: 'default'\n", __func__);
	LLAMA_LOG_WARN("%s: \n", __func__);
	LLAMA_LOG_WARN("%s: ************************************ \n", __func__);
	LLAMA_LOG_WARN("%s: GENERATION QUALITY WILL BE DEGRADED! \n", __func__);
	LLAMA_LOG_WARN("%s: CONSIDER REGENERATING THE MODEL \n", __func__);
	LLAMA_LOG_WARN("%s: ************************************ \n", __func__);
	LLAMA_LOG_WARN("%s: \n", __func__);
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_DEFAULT;
	} else if (
	tokenizer_pre == "default") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_DEFAULT;
	} else if (
	tokenizer_pre == "llama3" ||
	tokenizer_pre == "llama-v3" ||
	tokenizer_pre == "llama-bpe") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_LLAMA3;
	} else if (
	tokenizer_pre == "deepseek-llm") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_DEEPSEEK_LLM;
	} else if (
	tokenizer_pre == "deepseek-coder") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_DEEPSEEK_CODER;
	} else if (
	tokenizer_pre == "falcon") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_FALCON;
	} else if (
	tokenizer_pre == "mpt") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_MPT;
	} else if (
	tokenizer_pre == "starcoder") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_STARCODER;
	} else if (
	tokenizer_pre == "gpt-2" ||
	tokenizer_pre == "jina-es" ||
	tokenizer_pre == "jina-de" ||
	tokenizer_pre == "jina-v2-es" ||
	tokenizer_pre == "jina-v2-de") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_GPT2;
	} else if (
	tokenizer_pre == "refact") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_REFACT;
	} else if (
	tokenizer_pre == "command-r") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_COMMAND_R;
	} else if (
	tokenizer_pre == "qwen2") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_QWEN2;
	} else if (
	tokenizer_pre == "olmo") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_OLMO;
	} else if (
	tokenizer_pre == "dbrx") {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_DBRX;
	} else {
	throw std::runtime_error(format("unknown pre-tokenizer type: '%s'", tokenizer_pre.c_str()));
	}
	} else {
	vocab.type_pre = LLAMA_VOCAB_PRE_TYPE_DEFAULT;
	}

[teleprint-me]

@ggerganov Thats not necessary. I solved this already in #7219 and ##7117.

[turian]

Hi @BramVanroy I was encouraging you in #7022 to test that HF and llama tokenization are identical. Here is a colab you could modify to try: https://colab.research.google.com/drive/1RYlEj2UhylYWyaASFo-LLATzZ8d29Z0T?usp=sharing

[BramVanroy]

I'm unsure what has changed but it seems that phi-2 models are working again so that's good news. Will close this one for now.

[RhinoDevel]

I'm unsure what has changed but it seems that phi-2 models are working again so that's good news. Will close this one for now.

Well, convert-hf-to-gguf-update.py still doesn't have a "phi-2" entry. The models should work with the default tokenizer, though (but that is the case for a long time).