Code

tokenizer training script#103

Code

Merged

jettjaniak merged 8 commits intomaindelphi-suite/delphi:mainfrom tokenizer_trainingdelphi-suite/delphi:tokenizer_trainingCopy head branch name to clipboardApr 19, 2024

+179-23

Commits

Commits on Apr 19, 2024

• draft for tokenizer training script

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  31b149a

  Copy full SHA for 31b149a

• Update scripts/train_tokenizer.py

  Show description for 6c84224
  

  jannik-brinkmann

  and

  jettjaniak

  committed
  6c84224

  Copy full SHA for 6c84224

• integrate some of the suggested changes (WIP)

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  623f358

  Copy full SHA for 623f358

• another use of tempfile (untested)

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  dff911b

  Copy full SHA for dff911b

• bug fixes and test script

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  4fc7757

  Copy full SHA for 4fc7757

• split the code, reduced num. tmp files, ...

  jettjaniak

  committed
  e131120

  Copy full SHA for e131120

• removed tests/scripts

  jettjaniak

  committed
  1014d6b

  Copy full SHA for 1014d6b

• updated transformers to 4.40.0

  jettjaniak

  committed
  6d0ace0

  Copy full SHA for 6d0ace0