Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support loading vocab from fast tokenizer config in convert.py #3633

Merged
merged 32 commits into from
Dec 14, 2023
+162 −150
Merged

support loading vocab from fast tokenizer config in convert.py #3633

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
f7e377d
Add HFVocab into convert.py
strutive07 Oct 15, 2023
b0e00cb
Update convert.py
strutive07 Oct 15, 2023
f888d2e
Update convert.py
strutive07 Oct 15, 2023
ea9f35f
add bytes_to_unicode function
strutive07 Oct 15, 2023
c7b636e
change add_meta_vocab fucntion
strutive07 Oct 15, 2023
6ec856b
remove debug code
strutive07 Oct 15, 2023
1f16e5f
remove byte_encoder
strutive07 Oct 15, 2023
e876aec
Add newline between classes
strutive07 Oct 15, 2023
1778450
Check tokenizer.json when tokenizer.model is not exist.
strutive07 Oct 15, 2023
a5b26b6
Move transformers dependency to local code
strutive07 Oct 18, 2023
5a1f178
Add error context with 'raise from'
strutive07 Oct 18, 2023
89611cb
Add fast tokenizer option to BpeVocab
strutive07 Oct 23, 2023
97f690a
Merge branch 'master' into convert_hf_vocab
strutive07 Oct 29, 2023
e715442
Update convert.py
strutive07 Oct 29, 2023
d54764d
Add VocabLoader and remove *Vocab class
strutive07 Oct 30, 2023
e19b780
Add transformers dependency
strutive07 Oct 30, 2023
28f09be
remove added tokens and check newline token to decide spm or bpe
strutive07 Nov 5, 2023
4adb8b9
Update convert.py
strutive07 Nov 5, 2023
13f0701
Add special token type
strutive07 Nov 5, 2023
f37a7d7
Update convert.py
strutive07 Nov 11, 2023
9f4dc23
Update convert.py
strutive07 Nov 11, 2023
dcf372e
Update convert.py
strutive07 Nov 11, 2023
cc1f3fc
Fix typo in convert.py
strutive07 Nov 15, 2023
026eb7c
Fix when params.n_vocab < tokenizer vocab size
strutive07 Nov 18, 2023
2e263ca
update vocab class
strutive07 Nov 19, 2023
5ac1949
change funtion name
strutive07 Nov 22, 2023
74d80a8
Merge branch 'master' into convert_hf_vocab
strutive07 Nov 22, 2023
61edd1b
Remove unused variable/functions, add types to class variable and met…
strutive07 Nov 28, 2023
1f5357c
fix flake8 warnings
strutive07 Nov 28, 2023
8fabb01
code style cleanup
cebtenzzre Dec 13, 2023
c3b1c12
make mypy happy
cebtenzzre Dec 13, 2023
35e95b6
change exception
strutive07 Dec 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading