v0.2.2

What's Changed

• [BUG] Fix path to support windows machines by @not-lain in #12
• support local tiktokenizer loading by @not-lain in #13
• Fix #10: Add local tokenizer support by @bhavnicksm in #16
• Bump version to 0.2.2 and update init.py to reflect the new version by @bhavnicksm in #17

New Contributors

• @not-lain made their first contribution in #12

Full Changelog: v0.2.1...v0.2.2

v0.2.1

What's Changed

• Add support for WordPiece Tokenizers + Aya Family by @bhavnicksm in #9

Full Changelog: v0.2.0...v0.2.1

v0.2.0

What's Changed

• Remove tokenizers dependency from AutoTikTokenizers by @bhavnicksm in #8

Full Changelog: v0.1.2...v0.2.0

v0.1.2

What's Changed

• Update README.md by @bhavnicksm in #5
• Update README.md by @bhavnicksm in #6
• Bump version to 0.1.2 and update package metadata by @bhavnicksm in #7

Full Changelog: v0.1.1...v0.1.2

v0.1.1

What's Changed

• Bump version to 0.1.1 and enhance _AutoTikTokenizer functionality by @bhavnicksm in #4

Full Changelog: v0.1.0...v0.1.1

v0.1.0

What's Changed

• Add logo + Bump version to v0.1.0 by @bhavnicksm in #3

Full Changelog: v0.0.1a0...v0.1.0

v0.0.1a0

What's Changed

• Initial Commit by @bhavnicksm in #1
• Add version and author metadata to AutoTikTokenizer by @bhavnicksm in #2

New Contributors

• @bhavnicksm made their first contribution in #1

Full Changelog: https://github.com/bhavnicksm/autotiktokenizer/commits/v0.0.1a0