Code

feat-resize-tokenizer-add-new-tokens#9

Code

Open

Erland366 wants to merge 1 commit intounslothai:mainunslothai/unsloth-zoo:mainfrom Erland366:feat/resize-tokenizer-add-new-tokenErland366/unsloth-zoo:feat/resize-tokenizer-add-new-tokenCopy head branch name to clipboard

+35-3

Commits

Commits on Oct 31, 2024

• feat: enhance add_new_tokens function with resizing option and improved validation

  Erland366

  committed
  0a07389

  Copy full SHA for 0a07389