Code

Adding pretty print of tokenizer#1540

Code

Closed

haixuanTao wants to merge 3 commits intohuggingface:mainhuggingface/tokenizers:mainfrom haixuanTao:mainhaixuanTao/tokenizers:mainCopy head branch name to clipboard

+24-14

Commits

Commits on May 23, 2024

• Add pretty print formating of tokenizer when printed

  haixuanTao

  committed
  f9841b2

  Copy full SHA for f9841b2

• Fix rust-toolchain version

  haixuanTao

  committed
  ffdc8c3

  Copy full SHA for ffdc8c3

• Fix clippy warnings

  haixuanTao

  committed
  dc5b9bb

  Copy full SHA for dc5b9bb