Add Terminal-Based Visualization Tool for Tokenized Data Points in Tiktoken Tokenizer

[LVivona]

Key Features

• Token Visualization: Display token, and their positions in the input text.
• Interactive Interface: Allows users to input text and see the tokenized output in real-time.
• Enhanced Debugging: Facilitates better understanding of tokenizer behavior and helps in identifying issues.

Benefits

• Improved Usability: Provides a straightforward way to visualize and analyze tokenized data.
• Immediate Feedback: Users can quickly see the effects of changes to the tokenizer.
• Educational: Helps users grasp model information and the tokenization process more effectively.

import tiktoken
enc = tiktoken.get_encoding("gpt2")
enc.environment()

[LVivona]

Summary

Added current a terminal-based environment for visualizing tokenized data using Tiktoken. This feature provides immediate feedback on token types and positions, helping users better understand and debug tokenization.

Details:

• Using a popular rust tui framework ratatui for rendering terminal components.
• Introduced function environment in encoding class to launch an interactive tokenizer visualization.
• Displays token types and positions in the input text.
• Enhances debugging and comprehension of tokenized data points.