Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added token selector #92

Merged
merged 1 commit into from
Mar 30, 2024
Merged

Added token selector #92

merged 1 commit into from
Mar 30, 2024

Conversation

menamerai
Copy link
Collaborator

Adds the function to make token selector from tokenizer's vocab. Basically, the input should be a dict[str, int] that you can get from calling tokenizer.get_vocab(). We could also change the input to the tokenizer instead, but it's like that for now.

The function returns two thing, the first thing is the actual widget itself. If you want the string tokens from the widget, you can call widget.value. The second thing is a list of tokens ids that correspond to the selected tokens that are continually updated.

@menamerai menamerai added the eval label Mar 30, 2024
@menamerai menamerai self-assigned this Mar 30, 2024
@jettjaniak jettjaniak merged commit 01eb277 into main Mar 30, 2024
1 check passed
@jettjaniak jettjaniak deleted the token-select-gui branch March 30, 2024 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants