Possible bug with raw string handling

[DavisVaughan]

> r"(foo(arg = "\\dots", ...))"
[1] "foo(arg = \"\\\\dots\", ...)"


> treesitter::text_parse('r"(foo(arg = "\\dots", ...))"', treesitter.r::language())
<tree_sitter_node>

── Text ──────────────────────────────────────────────────────────────────────────────────────────────────────
r"(foo(arg = "\dots", ...))"

── S-Expression ──────────────────────────────────────────────────────────────────────────────────────────────
(program [(0, 0), (0, 28)]
  (identifier [(0, 0), (0, 1)])
  (string [(0, 1), (0, 14)]
    "\"" [(0, 1), (0, 2)]
    content: (string_content [(0, 2), (0, 13)])
    "\"" [(0, 13), (0, 14)]
  )
  (ERROR [(0, 14), (0, 15)]
    "\\" [(0, 14), (0, 15)]
  )
  (identifier [(0, 15), (0, 19)])
  (string [(0, 19), (0, 28)]
    "\"" [(0, 19), (0, 20)]
    content: (string_content [(0, 20), (0, 27)])
    "\"" [(0, 27), (0, 28)]
  )
)

[DavisVaughan]

Odd minimal reprex

works

treesitter::text_parse('r"( () )"', treesitter.r::language())
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> r"( () )"
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (program [(0, 0), (0, 9)]
#>   (string [(0, 0), (0, 9)])
#> )

works

treesitter::text_parse('r"(() )"', treesitter.r::language())
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> r"(() )"
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (program [(0, 0), (0, 8)]
#>   (string [(0, 0), (0, 8)])
#> )

doesnt work

treesitter::text_parse('r"(())"', treesitter.r::language())
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> r"(())"
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (program [(0, 0), (0, 7)]
#>   (identifier [(0, 0), (0, 1)])
#>   (string [(0, 1), (0, 7)]
#>     "\"" [(0, 1), (0, 2)]
#>     content: (string_content [(0, 2), (0, 6)])
#>     "\"" [(0, 6), (0, 7)]
#>   )
#> )

doesnt work

treesitter::text_parse('r"( ())"', treesitter.r::language())
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> r"( ())"
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (program [(0, 0), (0, 8)]
#>   (identifier [(0, 0), (0, 1)])
#>   (string [(0, 1), (0, 8)]
#>     "\"" [(0, 1), (0, 2)]
#>     content: (string_content [(0, 2), (0, 7)])
#>     "\"" [(0, 7), (0, 8)]
#>   )
#> )

so the trailing double )) is an issue?

[DavisVaughan]

Ah, got it. It's a bad interaction of calling advance() in the loop increment AND in the loop body. It really does only affect this weird case of back-to-back ) (maybeeee you can construct a case with - involved, not sure)

• Line 330: Hit first ), advance to 2nd )
• Line 351: ) is not a closing quote character, continue
• Loop increment: Advance to ", skipping handling of )!

tree-sitter-r/src/scanner.c

Lines 328 to 352 in a0d3e33

	for (; lexer->lookahead != 0; lexer->advance(lexer, false)) {
	// consume a closing bracket
	if (lexer->lookahead != closing_bracket) {
	continue;
	}
	lexer->advance(lexer, false);
	// consume hyphens
	bool hyphens_ok = true;
	for (int i = 0; i < hyphen_count; i++) {
	if (lexer->lookahead != '-') {
	hyphens_ok = false;
	break;
	}
	lexer->advance(lexer, false);
	}
	if (!hyphens_ok) {
	continue;
	}
	// consume a closing quote character
	if (lexer->lookahead != quote) {
	continue;
	}

I imagine that if we change this to a while loop and only increment in the body, it should work. Probably need to add an advance() in that first if (lexer->lookahead != closing_bracket) { branch.

[DavisVaughan]

A few more test cases

r"((\d+))" # -> yellow wave under the "r"
stringr::str_extract("foo123", r"((\d+))")  # -> red wave under the "r"