Fixed the issue of being unable to handle transformer added/expanded model tokens#83

Merged

noamgat merged 1 commit intonoamgat:mainfrom Qubitium:fix-tokenizer-size

Mar 10, 2024

+1-1

Commits on Mar 7, 2024

Fixed the issue of being unable to handle added/expanded model tokens. For transformers, tokenizer.vocab_size excludes all tokens added via token expansion. Correct usage here is len(tokenizer).
Qubitium
committed