Skip to content

Commit

Permalink
Add huggingface links
Browse files Browse the repository at this point in the history
  • Loading branch information
finetunej authored Aug 2, 2023
1 parent 487ad01 commit 2b98f1a
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ The JavaScript implementation used by the NovelAI frontend can be found [here](h

For V2, the original digit special tokens were replaced with english contractions. Digits will therefore be encoded using corresponding the byte tokens instead.

## Huggingface

Our tokenizer is also available on Huggingface Hub:

* https://huggingface.co/NovelAI/nerdstash-tokenizer-v1
* https://huggingface.co/NovelAI/nerdstash-tokenizer-v2

## License

The tokenizer is licensed under the GNU General Public License, version 2.

0 comments on commit 2b98f1a

Please sign in to comment.