Skip to content

Commit

Permalink
Update convert_hf_to_gguf.py
Browse files Browse the repository at this point in the history
Co-authored-by: compilade <[email protected]>
  • Loading branch information
MollySophia and compilade authored Aug 12, 2024
1 parent 9ba8fb6 commit 6edbe81
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion convert_hf_to_gguf.py
Original file line number Diff line number Diff line change
Expand Up @@ -2719,7 +2719,7 @@ def set_vocab(self):
for line in lines:
parts = line.split(' ')
assert len(parts) >= 3
_, token, token_len = int(parts[0]), ast.literal_eval(' '.join(parts[1:-1])), int(parts[-1])
token, token_len = ast.literal_eval(' '.join(parts[1:-1])), int(parts[-1])
token = token.encode("utf-8") if isinstance(token, str) else token
assert isinstance(token, bytes)
assert len(token) == token_len
Expand Down

0 comments on commit 6edbe81

Please sign in to comment.