Improve syntax highlighting using a Language Server Protocol implementation

[ggrossetie]

We currently rely on a TextMate grammar but it cannot properly parse AsciiDoc.

I think we should migrate to semantic highlighting: code.visualstudio.com/api/language-extensions/semantic-highlight-guide or LSP: code.visualstudio.com/api/language-extensions/language-server-extension-guide

If someone wants to give it a try, please feel free to do so!

Related issues:

• syntax highlighting bug with unix shell code sections such as [source,bash] #685
• Syntax highlighting spills out of source block #319
• Alternative code block breaks syntax hightlighting #439
• Inline Syntax Highlighting breakes with escaped ] #375
• Syntax Highlighting lost after using ListingBlock inside an OpenBlock #389
• Syntax highlighting broken by some JS source block #403
• Syntax highlighting broken for inline code and admonition blocks #342
• Unterminated #ifndef causes syntax highlighting to go wrong outside the source block #713
• syntax highlighter of source code block fails when using options #793
• Apostrophes (single quotes) triggering weird highlighting. #804
• Syntax highlighting broken if a variable is used with bash source #875
• Verse delimited block syntax breaks syntax highlighting. #893
• Section syntax highlighting stop working when adding extra space to a table separator (|== ) #894
• Incorrect highlight of sentence after footnote #930

[ggrossetie]

It's worth noting that semantic tokens do not support embedded languages:

microsoft/vscode#163292

In other words, source blocks won't be properly highlighted in AsciiDoc.

I don't think we should adopt the semantic tokens feature until embedded languages are supported.

I guess our only solution right now is to use LSP.

[ViToni]

Is this basically the same issue as #625 ?

[ggrossetie]

Initially, I thought it would be possible to implement a syntax highlighter using semantic token provider, that's why I've created a new issue.

Since we need to use LSP, it's indeed a duplicate of #625. I think we should close #625 in favor of this one.

[ViToni]

Looking into LSP has been on my ToDo list for a long time. This issue might be somethIng to get this action item checked off. Wondering though if it might be better to have a dedicated project for that as a LSP implementation (e.g. asciidoctor-lsp) could be used independently of this extension.

[ggrossetie]

@ViToni If you want to work on it that would be awesome! Yes, we should create a dedicated project! In case you've missed it, the AsciiDoc specification is under way:

• Community chat: https://asciidoc.zulipchat.com/
• AsciiDoc Lang Repository: https://gitlab.eclipse.org/eclipse/asciidoc-lang/asciidoc-lang/

The current syntax highlighter is broken and we could do so much more with a proper LSP (suggestion, lint, reformat and so on)

[ViToni]

@ggrossetie I'd try to get my feets wet. Not sure how far I get, but at least I'd like to give it a try.
I'll join the chat in the next days.

[ViToni]

Related to: asciidoctor/asciidoctor#3630

[ggrossetie]

@ViToni I'm starting to experiment a bit with LSP and semantic tokens. I think the first step is to establish a list of token types and token modifiers.

As far as I understand, LSP relies on semantic tokens to do syntax highlighting : https://microsoft.github.io/language-server-protocol/specifications/specification-3-16/#textDocument_semanticTokens

[ggrossetie]

It's worth noting that semantic tokens do not support embedded languages:
microsoft/vscode#163292

Having said that, it's possible to mix semantic tokens and TextMate grammars. The trick is to suppress semantic tokens for source block.

@ViToni if you are still interested, I think we should focus on writing a semantic token provider. We can then port this code over to the server (LSP).

[ggrossetie]

I think the first step is to establish a list of token types and token modifiers.

@ahus1 I guess you did pretty much the same things for the Intellij plugin? I found: https://github.com/asciidoctor/asciidoctor-intellij-plugin/blob/main/src/main/java/org/asciidoc/intellij/lexer/AsciiDocTokenTypes.java

Do you think it would make sense to share a common list of token types for syntax highlighting? Here's what I have so far:

• documentTitle
• documentAuthor
• documentAuthorEmail
• documentRevision
• attributeKey
• attributeValue
• attributeReference
• attributeSoftWrap
• attributeHardWrap
• listContinuation
• listMark
• blockAttrList
• blockTitle
• formattingText
• formattingMark

Token modifiers:

• constrained
• unconstrained
• strong
• emphasis
• code

In quite a few cases, an element will have both markup and text. For example, for the document title, = (markup) and Document title (text).

I don't know if we should use different token types? i.e., documentTitleMark for the token = and documentTitleText for the token Document title? Or just use documentTitle on = Document title?
Implementation wise, we could use the concept of modifiers to have documentTitle with modifier mark on = and documentTitle on Document title with the modifier text.

Regarding overlapping tokens, clients (IDE) might or might not support them:

<documentTitle><markup>=</markup> <text>Document title</text></documentTitle>

So for compatibility, it's probably safer to not use them if we can.