-
Notifications
You must be signed in to change notification settings - Fork 7
Scanner and contiguous sequence readers for symbolic expression lexer #29
Conversation
- We'll handle that in the Parser
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome! Just a few comments
} | ||
|
||
// Valid number parts are written to the buffer | ||
if isNumber(rn) || rn == '.' { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I understand correctly, this .3..14...1....5.....9.....
won't trigger an error?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@asdine you do understand it correctly - we'll validate that it really is a valid number when we do type checking in the parser.
Co-Authored-By: tealeg <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice! 💪👏👍
Co-Authored-By: tealeg <[email protected]>
This PR defines the scanner for the symbollic expression language to be deployed in the Regula UI.
There are essentially 3 layers to this:
Scanner
struct that wraps aio.Reader
in abufio.Reader
and uses it to read the stream a rune at a time (supporting backtracking as needed).readRune
andunreadRune
which wrap thebufio.Reader
methods of the same name - maintaining positional information as they go.Scanner.Scan
method grabs the next rune and identifies it. If it is a simple control character (i.e. parenthesis) it returns the token for that control character. If it implies a contiguous block, it hands this work off to a specialised scan function.The specialised scan functions make up the bulk of the work. Mostly they are simply looping looking for a character that terminates the sequence.
scanNumber
however is somewhat more complex. I have chosen to handle the conflict between the use of-
to indicate a negative number and it's use as an operator in this part of the code. This decision makes the scanner more complicated, but simplifies things in the parser we will add later - we have to make one of these two layers somewhat "impure" so I chose to do it at the more fundamental of the two, meaning the parser can be implemented without this wart.Fixes: #16