Langium playground error "Your grammar contains errors" when trying to use terminal TEXT: /[^<&]+/;

[dmarhas]

I am trying to create an XML parser with Langium but not able to define the terminal rule:

terminal TEXT: /[^<&]+/;

Are there any workarounds?

[msujew]

Hey @dmarhas,

the language server initialization fails due to a probably unwanted side effect of your terminal:

This terminal effectively tokenizes all input that isn't either < or &. However, since it also accepts (whitespace) Langium automatically puts it to the front of the terminal list, it will always hide all keywords and most other terminals as well. Aside from that technical detail, the terminal rule is way to eager.

You probably have a slight misconception of how LL parsers work. The tokenization phase is not context dependent, meaning that it won't start reading the TEXT terminal once it arrives at the parser rule that wants to read the TEXT terminal. Instead, it will go through the input once and tokenize everything according to the defined terminals. Only once it has all the tokens, it will start with the actual parsing phase.

To see how a chevrotain/Langium based lexer for XML should work, please take a look here.