Generating Illegal Escape Tokens

[jamesdborin]

When generating with LMFE the model is able to generate escape tokens like \n, and whenever it does this it becomes very slow. This sounds a lot like a previous issue: #41

Here is an example that I generated using this model: TheBloke/CapybaraHermes-2.5-Mistral-7B-AWQ

Prompt:

[INST]

Generate a summary full of new lines and new line characters.

[/INST]

which used the schema:

{"properties": {"summary": {"type": "string"}}, "required": ["summary"], "type": "object"}

Output:

{ "summary" : "A \nsummary full \nof new lines and \nnew line characters.\n\n" }  

I tested this on version 0.9.6 and 0.8.2 (a version released shortly after #41 was resolved).

[noamgat]

The output you are pasting here seems legal. Am I missing something? Perhaps there is a performance slowdown when the model does this, but I want to make sure that the output is legal.Where do you see illegal escape tokens?

…

On Fri, Apr 19, 2024 at 2:38 PM jamesdborin ***@***.***> wrote: When generating with LMFE the model is able to generate escape tokens like \n, and whenever it does this it becomes very slow. This sounds a lot like a previous issue: #41 <#41> Here is an example that I generated using this model: TheBloke/CapybaraHermes-2.5-Mistral-7B-AWQ Prompt: [INST] Generate a summary full of new lines and new line characters. [/INST] which used the schema: {"properties": {"summary": {"type": "string"}}, "required": ["summary"], "type": "object"} Output: { "summary" : "A \nsummary full \nof new lines and \nnew line characters.\n\n" } I tested this on version 0.9.6 and 0.8.2 (a version released shortly after #41 <#41> was resolved). — Reply to this email directly, view it on GitHub <#90>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKFA2BRDKA5WCCBTUCUMXTY6D62JAVCNFSM6AAAAABGPAPNYWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2TENZWGM2TMNQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[AriX]

I saw this slowdown in the past and it was caused by an edge case where JSONSchemaParser's shortcut_key implementation did not consider the correct parser while finishing parsing an escape sequence. When the UnionParser that StringParsingState pops onto the stack after getting a BACKSLASH is finished but not yet removed from the stack, shortcut_key fails to return json_freetext.

[noamgat]

Can you check if v0.9.8 is faster in this case?

[jamesdborin]

Whoops my mistake, I thought that /n was illegal - but yes part of the issue was this it was very slow when generating. Ill try the new version and see if that's faster.