From 2b98f1a44364087b52250c4e1c2a0159622ad42e Mon Sep 17 00:00:00 2001 From: finetune <82650881+finetunej@users.noreply.github.com> Date: Thu, 3 Aug 2023 01:30:18 +0200 Subject: [PATCH] Add huggingface links --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index 4bcba3e..658101e 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,13 @@ The JavaScript implementation used by the NovelAI frontend can be found [here](h For V2, the original digit special tokens were replaced with english contractions. Digits will therefore be encoded using corresponding the byte tokens instead. +## Huggingface + +Our tokenizer is also available on Huggingface Hub: + +* https://huggingface.co/NovelAI/nerdstash-tokenizer-v1 +* https://huggingface.co/NovelAI/nerdstash-tokenizer-v2 + ## License The tokenizer is licensed under the GNU General Public License, version 2.