- 使用：
//linux 平台
cd tokenizer-json
mkdir build
cd build && cmake ..
make
//移动tokenizer.json到文件夹中- 目前仅对qwen模型进行测试更多测试请等待
2.参考：
mnn-llm
json
-
Notifications
You must be signed in to change notification settings - Fork 0
zjkhahah/tokenizer-json
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
HuggingFace llm model tokenizer cpp read
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published