Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
tokenize : add --show-count (token) option (#8299)
This commit adds a new option to the tokenize example, --show-count. When this is set the total number of tokens are printed to stdout. This was added as an option as I was concerned that there might be scripts that use the output from this program and it might be better to not print this information by default. The motivation for this is that can be useful to find out how many tokens a file contains, for example when trying to determine prompt input file sizes for testing. Signed-off-by: Daniel Bevenius <[email protected]>
- Loading branch information