When to use encode/decode_special_tokens?

[novice03]

Hello, I am using a llama-2-chat model with exllamav2. My typical prompt is something like [INST] <<SYS>> You are a helpful assistant..<</SYS>> Instruction ...[/INST]. Since I am adding system prompt and instruction tokens, do I need to set encode_special_tokens and decode_special_tokens to True?

[turboderp]

For that prompt specifically you wouldn't need encode_special_tokens and decode_special_tokens, because the [INST] and <<SYS>> tags don't have special token IDs. I guess there's no easy way to know this stuff in advance. However, Llama2-chat's prompt format does use special tokens (BOS and EOS). The full prompt format for muiltiple rounds looks like:

<s>[INST] <<SYS>> You are a helpful assistant...<</SYS>> Tell me a joke.[/INST] Knock knock!</s><s>[INST] Who's there?[/INST] Banana.</s><s>[INST]Banana who? [/INST]....

<s> and </s> should be represented by the control tokens and not encoded as strings, which you could accomplish with encode_special_tokens or with the add_bos etc. flags.

[novice03]

Got it, thank you!

[w013nad]

I did some testing with the tokenizer yesterday and if you use the chatml format, the '<|im_start|>' encodes to 5 separate tokens if you do not encode special tokens. However, if you do use it, it properly encodes to 1 token. I also noticed that I started getting much better results when I started encoding it. I was using mistral dolphin