Code

Add XLNetTokenizer#1206

Code

Open

susnato wants to merge 6 commits intokeras-team:masterkeras-team/keras-hub:masterfrom susnato:xlnet-tokenizersusnato/keras-nlp:xlnet-tokenizerCopy head branch name to clipboard

+344

Commits

Commits on Aug 10, 2023

• pre-init only licenses added

  susnato

  committed
  b55691e

• tokenizer encode and decode done

  susnato

  committed
  9ef8875

• tests all green!

  susnato

  committed
  f1fe2d8

• style fix

  susnato

  committed
  983d93d

• tests fix

  susnato

  committed
  e03e9bb

Commits on Aug 12, 2023

• tokenizer now working with ds.map(...)

  susnato

  committed
  e431196