Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple EOS token ids + fix bugs in JsonSchemaParser #123

Merged
merged 3 commits into from
Jul 27, 2024

Conversation

aw632
Copy link
Contributor

@aw632 aw632 commented Jul 19, 2024

Two contributions:

  1. We modify the code so that it can support multiple EOS token ids, while also being backwards compatible. (This is necessary for some new models that have multiple stop tokens, otherwise the parser will not stop correctly.)
  2. JsonSchemaParser was not handling number type correctly (leading zeros, decimal points, exponents like 1e5). I added a few dozen new test cases to cover this complex functionality and modified the NumberParser class. Coding style was a bit messy, completely open to any changes on that front.

@noamgat
Copy link
Owner

noamgat commented Jul 23, 2024

Thank you very much for the PR! I will thoroughly review it in a couple of days and merge it in. It looks great from first glance.

@noamgat noamgat merged commit 075fc2c into noamgat:main Jul 27, 2024
1 check passed
@noamgat
Copy link
Owner

noamgat commented Jul 27, 2024

Thank you very much for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants