Strange/wrong error message? Need help!

[d-albrecht]

Hey guys! So, I'm trying to use Langium for the first time with a formal grammar that is tried and true for years now (using a different parser). This part shouldn't be the problem here, But Langium behaves differently than I expected, so could someone shed some light on what I'm doing wrong here:

I have distilled the problem down to this language definition (plus the white-space and comment hidden terminals):

entry Model:
    (binds+=Bind)*;
Bind: '@' field=FIELD id=Bid;
Bid: '#' id=NAT;
terminal NAT returns number: /[0-9]+/;
terminal FIELD: /[a-zA-Z0-9_ ]+/;

So any strings that start with "@" then some text, then "#" and then some (positive) number should be fine. But the following example isn't parsed:

@test#1234

with the error:

Expecting token of type 'NAT' but found '1234'.

Any ideas where I went wrong here? This isn't my first rodeo with formal grammars or parsers. But even though I've read most of the Langium documentation (at least all the grammar parts) I can't figure out myself what I'm doing wrong here. Even turning Bid into a fragment didn't help. I'm out of ideas. And I would like to keep the individual definitions apart because this is just a subset of the whole grammar in this project and even in-lining Bid into Bind didn't help either.

Thanks in advance.

[snarkipus]

Sounds a lot like this. Maybe flipping the order of your terminal rules will fix it. @msujew will know for sure.

[d-albrecht]

Nope, that didn't change anything. Thanks anyways!

[luan-xiaokun]

Overlapping terminal rules should be avoided according to #365 (reply in thread), and replace the definition of FIELD with

terminal FIELD: /[a-zA-Z_]\w*/;

should resolve the problem.

[d-albrecht]

I will look into that alternative. But the exact way you defined field would only accept strings that start with digits, then contain non-digits, and possibly end again in digits (plus anything that starts in a non-digit). A string like "4g5h6" would not work. Therefore, the general pattern of /[a-zA-Z0-9_ ]+/ seems to require more care for (un-)ambiguity. I'm convinced that what I have in mind can be translated to Langium somehow, I just need to plan this out more carefully and with respect to the used Lexer. Now, I know how the error message is meant to be interpreted and with that knowledge I'm sure I can make some more progress right now. So, thanks again for your help.

[luan-xiaokun]

A string like "4g5h6" would not work.

I'm kind of lost here, the second alternative of /[a-zA-Z_]\w*|([0-9]+[a-zA-Z_]+)+[0-9]*/ is /([0-9]+[a-zA-Z_]+)+[0-9]*/, whose first component will match 4g5h, with two repetitions of the pattern [0-9]+[a-zA-Z_]+, and the second component will match the last 6, so I think that this regex should be able to correctly capture the 4g5h6 token. (Please let me know if I made a mistake)

But the exact way you defined field would only accept strings that start with digits, then contain non-digits, and possibly end again in digits

This is not quite true, as the first alternative will match strings that start with a non-digit, and then accept both digits and non-digits.

Just to confirm, you do want the terminal rule FIELD to capture something like _1abc and 1a2b3, but not pure numbers, right? I remember I tried this method and it works, you can give it a try. If there are tokens that it fails to capture, or accidentally captures, please let me know.

[d-albrecht]

Sorry, yes, you are completely right. I must have copied your definition in parts and apparently missed one part at one point. I just typed in such an example and it suddenly immediately worked. So, although the definition is a bit convoluted, it perfectly works. EXCEPT - and that's what I should have seen instead of my previous criticism - the field names can never be actual numbers. Your definition enforces that at least one character of the name is any non-digit, which wasn't my intention. But to be honest, I'm not sure if this case (a field being an actual number) can even occur and all that are details that shouldn't bother me too much at this point. Your fix worked after all. If the grammar as a whole works in Langium I can come back and tweak each individual definition.

[luan-xiaokun]

It's all right, I sometimes make typos too. Glad I could help you!

[d-albrecht]

This might not be the cleanest solution in respect to the resulting ASTs, but I just replaced any reference to FIELD with FIELD | NAT and this enables actual numbers as fields. And so far, nothing else breaks after I changed this. I just have to keep that in mind when working with the constructed AST that the fields can be of type number instead of string in some cases and leading zeros will be lost. Oh, well, no solution is absolutely perfect. ;) Or perhaps I can refine this approach even further at some point.