Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] CharacterLevelParserConfig gets ignored by tokenenforcer #131

Open
laurens-gs opened this issue Aug 14, 2024 · 1 comment
Open

[bug] CharacterLevelParserConfig gets ignored by tokenenforcer #131

laurens-gs opened this issue Aug 14, 2024 · 1 comment

Comments

@laurens-gs
Copy link

laurens-gs commented Aug 14, 2024

The following setup in the TokenEnforcer initialization cause any custom CharacterLevelParserConfigs to be overriden with default values

config = CharacterLevelParserConfig(alphabet=tokenizer_data.tokenizer_alphabet)
parser.config = config

I think the approach should be to directly modify the alphabet attribute of the existing config to keep other existing configurations such as max array length.

@noamgat
Copy link
Owner

noamgat commented Sep 3, 2024

Thanks for raising this. PRs welcome, if not, I will try to get to it in a few days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants