Skip to content

Commit

Permalink
Update src/modalities/dataloader/create_packed_data.py
Browse files Browse the repository at this point in the history
Co-authored-by: Felix Stollenwerk <[email protected]>
  • Loading branch information
le1nux and flxst authored Dec 16, 2024
1 parent 88b1201 commit 373c99a
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/modalities/dataloader/create_packed_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ def _get_required_num_of_bytes_to_repr(int_to_get_repr: int) -> int:
Returns:
int: The number of bytes required to represent the integer.
"""
# we currently only supoprt token sizes of 1, 2 and 4 bytes, as implemented here:
# we currently only support token sizes of 1, 2 and 4 bytes, as implemented here:
# https://github.com/Modalities/modalities/blob/fix_char_bytes_indexation_mismatch/src/modalities/dataloader/dataset.py#L202
num_bytes = math.ceil(math.log2(int_to_get_repr) / 8)
if num_bytes == 1:
Expand Down

0 comments on commit 373c99a

Please sign in to comment.