Parse SQL keywords and identifiers?

[jennykwan]

I'm writing a SQL parser, starting with a lexer, and am having trouble at basic keywords and identifiers.

1. Both keywords and identifiers begin with a single alphabetic character (Unicode, case insensitive) or an underscore, followed by zero or more Unicode alphabetic chars, underscores, ascii digits (0-9), and dollar signs. I'm having trouble putting the parser for just this together.

2. Next is perhaps a Rust question, but I want separate parsers for keywords and for identifiers. I want any keyword/identifier substring to match against a list of keywords in a case insensitive way. If it matches nothing, it should return an Err. In my attempt thus far, I use match and _ => return Err(), which then forces the rest of the match expression to be wrapped in an Ok. Does this work?

3. Finally, I don't know how to approach word boundaries. For example:

   select 1;

   should parse to Keyword::Select followed by Constant::Numeric("1"). The word boundary is whitespace.

   select(1);

   should parse to Keyword::Select, Special::LeftParenthesis, Constant::Numeric("1"), and Special::RightParenthesis. The word boundary is the state transition from t to (.

   I'm mimicking PostgreSQL where this is valid syntax.

Sorry; I'm a total newb and I've never worked with scanners or parsers before.

[jennykwan]

Here's what I have for the keyword parser:

#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
pub enum Keyword {
    Alter,
    And,
    ...
    Where,
}

pub fn keyword<'src>(
) -> impl Parser<'src, &'src str, (SimpleSpan<usize>, Keyword), extra::Err<Rich<'src, char, SimpleSpan>>>
{
    any()
        .filter(|c: &char| c.is_alphabetic() || *c == '_')
        .then(
            any()
                .filter(|c: &char| c.is_alphabetic() || *c == '_' || c.is_ascii_digit())
                .repeated()
                .at_least(0),
        )
        .to_slice()
        .map(|ident: &str| {
            Ok((match ident.to_uppercase().as_str() {
                "ALTER" => Keyword::Alter,
                "AND" => Keyword::And,
                ...
                "WHERE" => Keyword::Where,
                _ => return Err(()),
            },))
        })
}

This is failing the type checker, both in terms of the Result generated by the Ok, and missing span info.

What do I do instead?

[jennykwan]

Figured out the keyword parser:

pub struct Spanned<T>(T, SimpleSpan<usize>);

pub fn keyword<'src>(
) -> impl Parser<'src, &'src str, Spanned<Keyword>, extra::Err<Rich<'src, char, SimpleSpan>>> {
    any()
        .filter(|c: &char| c.is_alphabetic() || *c == '_')
        .then(
            any()
                .filter(|c: &char| c.is_alphabetic() || *c == '_' || c.is_ascii_digit())
                .repeated()
                .at_least(0),
        )
        .to_slice()
        .try_map_with(|ident: &str, extra| match ident.to_uppercase().as_str() {
            "ALTER" => Ok(Spanned(Keyword::Alter, extra.span())),
            "AND" => Ok(Spanned(Keyword::And, extra.span())),
            ...
            "WHERE" => Ok(Spanned(Keyword::Where, extra.span())),
            _ => return Err(Rich::custom(extra.span(), "expected keyword")),
        })
}

Working on word boundaries.

[Zij-IT]

I may be missing something, but is there a reason that chumsky::text::keyword doesn't work here?

It would then be:

keyword()
    .try_map_with(|kw| match kw.to_uppercase().as_str() {
        "SELECT" => Kw::Select,
        /* the rest */
        _ => Err(..),
    })

[zesterer]

My personal view is that the simplest way to set up keyword and identifier parsing looks something like this:

pub enum Token<'src> {
    KwSelect,
    KwAlter,
    ...
    Ident(&'src str),
}

text::ident().map(|x| match x {
    x if x.eq_ignore_ascii_case("select") => Token::KwSelect,
    x if x.eq_ignore_ascii_case("alter") => Token::KwAlter,
    ...
    x => Token::Ident(x),
})

This parses any C-style identifier, then checks to see whether it is a lower-case match with any of the given keywords, mapping it to the relevant keyword token if so. If not, it'll fall back to the identifier token.

If you really want other cases to fall back to an error, try_map can be used instead of map as mentioned by @Zij-IT, although I can't really see this being useful in practice.

[jennykwan]

Thanks! I can't get a clear answer online anywhere whether SQL identifiers, at least in PostgreSQL, are C-style identifiers as in Unicode Standard Annex 31. I'm actually using a different parser hacking together my own Unicode set; I'll have to come back to it. Accepting @zesterer's answer.

I think the "word boundary" question I had is actually a question of how to get chumksy to do backoff. I'll open a separate discussion for that.