Lexer design?

[prologic]

Something i noticed while trying to figure out how to add basic support for escape sequences is the way the Lexer is designed, looks like you're using recursion?

For example:

tau/lexer/lexer.go

Lines 142 to 151 in 6b87b87

	func lexString(l *lexer) stateFn {
	if l.peek() == '"' {
	l.emit(item.String)
	l.next()
	l.ignore()
	return lexExpression
	}
	l.next()
	return lexString
	}

Is there a risk of blowing up the Go stack here or has Golang itself become quite elver here and uses tail recursion? 😅 (I haven't tested this lately!)

[prologic]

Actually re-reading this is may not be recursion at all. I might be mistake, this looks like a state machine?

[NicoNex]

Yup, it's a state machine, it's very similar to the lexer that Go uses.

I really recommend watching the video Lexical Scanning in Go by Rob Pike both to know more and because the video itself is awesome. 😄

[NicoNex]

But anyway for the escape sequences iirc there's a function in the Go standard library for that.

[prologic]

O'really? Do tell!

[NicoNex]

I searched and I couldn't find it, so I probably was wrong :/.

[prologic]

Can you take a look at my WIP branch and see if I'm going in the right direction fo rthis?

[NicoNex]

I think that another option that you could consider is to add the string parsing in ast/string.go.
In that way you don't need to touch the lexer at all and it should be way easier.
Also I was thinking of using l.pos of the lexer to produce better error messages in the future, so I wouldn't change the user input in the lexer itself.

[prologic]

Ahh that actually makes sense, I'll do that instead 👌