Tab/space parsing issue

[Abhiroop]

As evidenced in this commit: d3b3a69

Extraneous tabs/spaces result in the following incomprehensible error:

ParseErrorBundle {bundleErrors = TrivialError 104 (Just (Tokens ('=' :| ""))) (fromList [Tokens ('&' :| "&"),Tokens ('(' :| ""),Tokens ('(' :| ")"),Tokens ('*' :| ""),Tokens ('+' :| ""),Tokens ('-' :| ""),Tokens ('/' :| ""),Tokens (';' :| ""),Tokens ('<' :| ""),Tokens ('=' :| "="),Tokens ('>' :| ""),Tokens ('F' :| "alse"),Tokens ('T' :| "rue"),Tokens ('|' :| "|"),Label ('d' :| "igit"),Label ('i' :| "nteger"),Label ('l' :| "owercase letter"),Label ('u' :| "ppercase letter")]) :| [], bundlePosState = PosState {pstateInput = "chan : Channel Int\n;chan = channel ()\n\n;foo : ()\n;foo =\n  let v = sync (recv chan) in\n  foo\n      \nmain =\n  let _ = spawnExternal chan 0 in\n  foo;", pstateOffset = 0, pstateSourcePos = SourcePos {sourceName = "testcases/good14.cam", sourceLine = Pos 1, sourceColumn = Pos 1}, pstateTabWidth = Pos 8, pstateLinePrefix = ""}}

Can we catch this and emit a more friendly error message?

[Rewbert]

I am quite confident that there's an issue in the preprocessor, especially when it comes to comments. I am guessing it's related. The preprocessor is quite greedy in its approach.

[Rewbert]

nextToken :: PP (T.Text, Int)
nextToken = do
    (ST t i ti) <- get

    -- first split the input up in its leading whitespace/newline and the rest
    let (fluff, t')  = T.span (\c -> c == ' ' || c == '\n') t
    -- then extract the next token and the rest of the input
    let (token, t'') = T.span (\c -> c /= ' ' && c /= '\n') t'


    -- update the internal column counter
    updateColumn fluff
    c <- gets current

    -- emit the things we just cut off, to keep
    -- the annotated code as close to the source as possible
    tell fluff

    -- modify the internal state to reflect the rest of the input
    modify $ \(ST _ c i) -> ST t'' c i

    -- get the column of the current token and return the subsequent pair
    column <- gets current
    return (token, column)

Here it seems as if I am only considering spaces and newlines. If there's a tab in the commit you referenced, this is probably where the issue is. The good thing about the preprocessor is that it's quite short and simple, so changing it should not be too difficult, but a bad thing is that I've not documented it very well. @Abhiroop

[Abhiroop]

Here it seems as if I am only considering spaces and newlines.

Aha! Perhaps that's the fix! Good that you documented it here. This is nothing very urgent; we can push this change later.

[Abhiroop]

This is not strictly a space/tab issue but this program:

main =
  let m = 3
  m

issues the following error:

ParseErrorBundle {bundleErrors = TrivialError 22 (Just (Tokens (';' :| ""))) (fromList [Tokens ('&' :| "&"),Tokens ('\'' :| ""),Tokens ('(' :| ""),Tokens ('(' :| ")"),Tokens ('*' :| ""),Tokens ('+' :| ""),Tokens ('-' :| ""),Tokens ('/' :| ""),Tokens ('<' :| ""),Tokens ('=' :| "="),Tokens ('>' :| ""),Tokens ('F' :| "alse"),Tokens ('T' :| "rue"),Tokens ('_' :| ""),Tokens ('i' :| "n"),Tokens ('|' :| "|"),Label ('d' :| "igit"),Label ('i' :| "nteger"),Label ('l' :| "etter"),Label ('l' :| "owercase letter"),Label ('u' :| "ppercase letter")]) :| [], bundlePosState = PosState {pstateInput = "main =\n  let m = 3\n  m;", pstateOffset = 0, pstateSourcePos = SourcePos {sourceName = "testcases/good16.cam", sourceLine = Pos 1, sourceColumn = Pos 1}, pstateTabWidth = Pos 8, pstateLinePrefix = ""}}

The in in let is missing. A more readable error message would help.