Skip to content
This repository has been archived by the owner on Nov 13, 2024. It is now read-only.

Commit

Permalink
Update docs/library.md
Browse files Browse the repository at this point in the history
Co-authored-by: igiloh-pinecone <[email protected]>
  • Loading branch information
acatav and igiloh-pinecone authored Oct 22, 2023
1 parent 2a51a30 commit b33eb16
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/library.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ By default, the global tokenizer is initialized with `OpenAITokenizer` that is b

<details>
<summary>Go deeper</summary>
The global tokenizer is holding an underlying tokenizer that implements `BaseTokenizer`.
The `Tokenizer` singleton is holding an inner `Tokenizer` object that implements `BaseTokenizer`.

You can add a customized tokenizer of your own by implement a subclass of `BaseTokenizer` and pass the class in the `tokenizer_class` parameter.

Expand Down

0 comments on commit b33eb16

Please sign in to comment.