how to extend tokenizer for LLaMa-3 ?

[StephennFernandes]

how did you guys extend the tokenizer in LLaMa-3 for chinese ? seems like unlike other llama models in the past this one is based off on a new tiktoken tokenizer. so how did you guys extend the tokenizer ?

[ymcui]

We did not extend vocabulary in this repository.

Ref: https://github.com/ymcui/Chinese-LLaMA-Alpaca-3/blob/main/README_EN.md#-uses-the-original-llama-3-vocabulary