llama : restore prefix space in llama tokenizer#4081

Merged

cebtenzzre merged 2 commits intomasterfrom ceb/restore-prefix-space

Nov 15, 2023

+4-1

Commits on Nov 14, 2023

Revert "dont add space when using special tokens"
cebtenzzre
committed

Commits on Nov 15, 2023

do not add space prefix if the first token is special
cebtenzzre
committed