How to make a parser that continues past an invalid match?

[nomnomab]

Hi, I am enjoying this library and it is being very helpful in use with an LSP.

However, I am quite confused on some parsing in relation to that type of environment. For example, with the following code, I can easily chain a parser per section with or_nots and that works fine:

shader lit {
  extern { }
  stage name { }
  stage name { }
}

The problem arises when the user does any invalid match in a parser, such as during autocomplete:

shader lit {
  extern { }

  foo // <- example

  stage name { }
  stage name { }
}

I want to still match the entire shader scope with my parser, but if it hits a bad entry, then I lose everything past foo, and my shader Span no longer covers the entire shader scope.

Is there a way to essentially make a parser that will scan for what it requires, but also continue past invalid matches? Or even better for my case, if I can throw all the invalids into a Vec to debug later? That would make it easier to still collect the user's bad input as a token.

This is my current parser for the shader portion:

let any = select! { |span|
    EfxToken::Any(s) => (s, span)
};

// parses the definition of the shader
let definition = just(EfxToken::Shader)
    // name
    .ignore_then(any)
    .then(
        // these return `or_not` parsers
        parse_externs()
            .then(parse_stages())
            .delimited_by(just(EfxToken::Control('{')), just(EfxToken::Control('}')))
    )
    .map_with_span(|(name, (externs, stages)), span| Shader {
        span,
        name,
        externs,
        stages,
    })
    // can be missing
    .or_not();

Is there a general approach used with LSP parsing for these kinds of things?

Thanks!

[zesterer]

This is what the recover_with combinator is for: it allows you to specify a fallback behaviour (such as another parser) that gets used when an error is encountered, allowing the error to be recovered from and parsing to continue.

[nomnomab]

Good to know! I didn't think to use that. I ended up finding another way to do what I needed prior to your message, but if recover_with can replace it, that would be cool.

/// Takes a parser and uses a skip parser to ignore prior tokens
/// that do not match what the first requires.
fn skip_before<T>(
    skip  : impl Parser<EfxToken, EfxToken, Error = Simple<EfxToken>> + Clone,
    parser: impl Parser<EfxToken, T, Error = Simple<EfxToken>> + Clone
) ->
    impl Parser<EfxToken, T, Error = Simple<EfxToken>> + Clone
    where T: Debug
{
    skip
        .ignored()
        .repeated()
        .ignore_then(parser.clone())
}

Didn't even realize any() was a thing, nor ignored() somehow 😂 Definitely made some stuff easier.

[nomnomab]

I'm a bit confused on how recover_with works, and the example provided in the doc isn't being entirely clear to me.

If I have something like this and I wanted it to keep going even if not all 3 slots are filled, is this a case for recover_with?

// technique{} parser
technique default {
    // compile(..) parser
    compile(all, vs, name_vs);
    // compile(..) parser
    compile(all, fs, );
}

What kind of recovery strategy would work for this case in particular? Would I put it in the parse_compiles parser after the repeated? Or is it meant to be more "in" the children parsers themselves?

fn parse_compiles() -> impl Parser<EfxToken, Option<Compiles>, Error = Simple<EfxToken>> + Clone {
    parse_compile()
        .repeated() // should it go here?
        .map_with_span(|x, span| Compiles {
            span,
            compiles: x
        })
        .or_not()
}

fn parse_compile() -> impl Parser<EfxToken, Compile, Error = Simple<EfxToken>> + Clone {
    let any = select! { |span|
        EfxToken::Any(s) => (s, span)
    };
    
    let comma = just(EfxToken::Control(','));
    
    // or somewhere in here as this entire block could fail
    select! { |span|
        EfxToken::Any(s) if s == "compile" => (s, span)
    } 
        .ignore_then(just(EfxToken::Control('(')))
        .ignore_then(
            // platform
            any
            .then_ignore(comma.clone())
            // stage
            .then(any)
            .then_ignore(comma.clone())
            // entry
            .then(any)
        )
        .then_ignore(just(EfxToken::Control(')')))
        .then_ignore(just(EfxToken::Control(';')))
        .map_with_span(|((platform, stage), entry), span| Compile {
            span,
            platform,
            stage,
            entry_point: entry
        })
}

I'm also not entirely sure of how to satisfy the recovery parser requirements either. Seems that unless I give it [] (like skip_then_retry_until) it is never happy with my custom types. Is there an "in-depth" example of recovery patterns?

[zesterer]

A suitable strategy here might be 'keep going until a matching ), like so:

compile_kw.ignore_then(args
    .delimited_by(lparen, rparen)
    // Matches `( ... )`
    .recover_with(via_parser(any().and_is(rparen.not()).repeated().delimited_by(lparen, rparen))))

I've simplified some of the definitions so this is more akin to pseudocode, by the way.