provide more custom semantic token types/modfiers

[Techatrix]

LSP semantic tokens offer a predefined set of semantic token types and semantic token modifiers. ZLS is currently emitting mostly standard token types with relatively few exceptions. The problem is that the existing token types are not able to accurately represent the language features of Zig which hurts the advantages that semantic highlighting can offer over syntactic highlighting.

Custom semantic tokens could enable ZLS to eventually explore new features like the following:

• differentiate between mutable and non-mutable values Improve Semantic Token Type ziglang/vscode-zig#213
• differentiate between different "kinds" of builtin functions Different semantic highlighting for builtin functions #2104
• highlight code that may invoke safety-protected Undefined Behavior (e.g. highlight @enumFromInt for non-exhaustive enums but not for exhaustive enums, @intCast but not @truncate, + but not +|, etc.)
• highlight inline functions at the callsite
• highlight escape sequences inside string literals
• highlight format specifiers like "{s}"

This issue encompasses two important tasks:

• specify a set of token types and token modifiers that are tailored to Zig
• investigate how various editors deal with custom semantic tokens

As far as I can tell, rust-analyzer uses a lot of non standard tokens. Their approach may serve as good source of inspiration.

[arrufat]

• investigate how various editors deal with custom semantic tokens

Regarding this part, I have experience using @kakoune-lsp, which provides a straightforward approach. Here's my configuration, where face fields are faces defined in the Kakoune editor, which can be set to any combination of color/bold/italic/underline/double underline/curly underline, as shown here.