Code

Draft backtrack#1712

Code

Draft

ArthurZucker wants to merge 10 commits intomainhuggingface/tokenizers:mainfrom backtrackhuggingface/tokenizers:backtrackCopy head branch name to clipboard

+2,262-17

Commits

Commits on Jan 3, 2025

• initial-commit

  ArthurZucker

  committed
  ac46243

• update test

  ArthurZucker

  committed
  3e884ac

• update benches as well

  ArthurZucker

  committed
  0907a4d

• nits

  ArthurZucker

  committed
  dca4568

• add no pretokenizer bench

  ArthurZucker

  committed
  d334fb4

• push latest changes

  ArthurZucker

  committed
  a3ed0c3

Commits on Jan 4, 2025

• updates

  ArthurZucker

  committed
  7c9e534

• nits

  ArthurZucker

  committed
  ee18ba9

Commits on Jan 6, 2025

• update serialization to support initializing from BPE

  ArthurZucker

  committed
  4b63a7a

• nits

  ArthurZucker

  committed
  a7baf1b