New tokenizers support

[dianjixz]

Do you have plans to use the new 的 tokenizers? Transformers are too slow; it takes 10 seconds to start on ax630C. If possible, could you refer to the implementation at: https://github.com/huggingface/tokenizers.git? This one imports very quickly.

[dianjixz]

I tried to write a tokenizer service, which might be useful. You can find it at: https://github.com/m5stack/StackFlow/blob/dev/projects/llm_framework/main_llm_tokenizer/_tokenizer.py