How to handle Tokens when the lexical grammar might be non regular

[vikigenius]

I am working on a toy CSS parser and it seems to have really complicated identifiers that don't seem like they can be captured by a regex?

https://www.w3.org/TR/css-syntax-3/#ident-token-diagram

And also what if you have to treat a Token differently based on surrounding context? Is it an identifier or an arithmetic operation (CSS/SASS allow hyphens in their identifier)?

How would I handle such cases in Logos? Would you recommend just combining the parser and lexer for cases like these and not use Logos?

[jeertmans]

Hi @vikigenius, this is probably possible using extras (Extra) or callbacks, did you check the examples and the book?

Context-based lexing is pretty much a matter of you implementing the necessary logic inside callbacks.