Skip to content

Commit

Permalink
Update scripts/train_tokenizer.py
Browse files Browse the repository at this point in the history
Co-authored-by: Jett <[email protected]>
  • Loading branch information
jannik-brinkmann and jettjaniak authored Apr 5, 2024
1 parent ad75acf commit 1b799fc
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions scripts/train_tokenizer.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,9 @@
import os

from datasets import load_dataset
from tqdm.auto import tqdm
from tqdm.auto import tqdm, trange
from transformers import LlamaTokenizerFast
from tokenizers import SentencePieceBPETokenizer
from tqdm import trange

from delphi.train.tokenizer import train_vocab, get_tokenizer_model_path

Expand Down

0 comments on commit 1b799fc

Please sign in to comment.