Fixed the issue of being unable to handle transformer added/expanded model tokens

[Qubitium]

For transformers, tokenizer.vocab_size excludes all tokens added via token expansion. Correct usage here is len(tokenizer).

ref: https://stackoverflow.com/questions/67412925/what-is-the-difference-between-lentokenizer-and-tokenizer-vocab-size
ref: huggingface/tokenizers#900 (comment)

Without this PR, any new custom tokens added to the transformer model and subsequently trained will be invisible to the lm-format-enforcer.

[Qubitium]

@turboderp my pr fix only fixed tranformer tokenizer integration as I am unfamiliar with exllama tokenizer. Perhaps the same patch is also required for exllama integration depending how the exllama tokenizer normalize "vocab size"? I find the transformer current discrepancy a little strange.

[Qubitium]

@noamgat Please review this bug fix. Thanks.

[noamgat]

Thanks for the contribution!

[JoshC8C7]

Just a warning that this change now prohibits using models whose vocab size (normal + added tokens) is larger than its model's embedding size (for a model which hasn't been retrained to output the added tokens). Not a common case (although it is my own, as I've got extra post-tokenization-pre-inference steps) but worth noting nonetheless.