Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preserving original token location throughout tokenization phase #181

Open
declard opened this issue Nov 12, 2024 · 0 comments
Open

Preserving original token location throughout tokenization phase #181

declard opened this issue Nov 12, 2024 · 0 comments

Comments

@declard
Copy link

declard commented Nov 12, 2024

I've split out the lexer phase to avoid dealing with whitespaces and comments at the grammar level, so now there is a Pidgin parser for String->[Token] and a Pidgin parser for [Token]->AST.

How do I wire the original token locations correctly?

So far I found the SourcePosDelta/SourcePos types with CurrentSourcePosDelta/CurrentPos parsers, and the WithPosCalculator ext method. My hope was that I could plug SourcePosDelta in it using WithPosCalculator(token => token.Position), but the implementation treats it as the delta between the tokens (something like a token length, but not quite), not the delta from the beginning of the stream/enumerable/whatever-the-source-is.

Is my only option to compute token "lengths"?

If I make a PR that adds WithPosProvider ext method and

if (_sourcePosProvider is not null)
{
    var current = (int)(location - _bufferStartLocation);
    return _sourcePosProvider(_span[current]);
}

may I hope that it's accepted?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant