x/tools/gopls: semantic tokenizing of string escape/format characters

[soypat]

What version of Go are you using (go version)?

$ gopls version
golang.org/x/tools/gopls v0.6.10
    golang.org/x/tools/[email protected] h1:8Ebz8PymS2umcuCFhoz67unyJfWsipjTIrkBUF9kypg=

$ VsCode settings:
 "gopls": {	
        "ui.semanticTokens": true,
	},
    "editor.semanticHighlighting.enabled": true,

What did you do?

Write following code in VSCode with semanticTokens enabled

const httpResponse = "HTTP/1.0 200 OK\r\nContent-Type: text/html\r\nPragma: no-cache\r\n\r\n<h2>..::TinyGo Rocks::..</h2>"
fmt.Sprintf("%v", httpResponse)

What did you expect to see?

Semantic tokenization of \n\r\t escaped characters and format characters like %v in strings inside a Format function.

What did you see instead?

Monocolor string like the one github displays.

[pjweinb]

What should semantic tokens return in this case? (the possibilities are in the section on semantic tokens in https://microsoft.github.io/language-server-protocol/specifications/specification-3-17/)

[soypat]

Maybe an unused token, such as class could be used for such end. I'm not sure how to check what VScode uses for other languages, is there a way?

From here the following semantic tokens are available:

Semantic Tokens

• type
• class
• enum
• struct
• typeParameter
• parameter
• variable
• property
• enumMember
• event
• function
• method
• macro
• keyword
• modifier
• comment
• string
• number
• regexp
• operator

Semantic Token Modifiers

• declaration
• definition
• readonly
• static
• deprecated
• abstract
• async
• modification
• documentation
• defaultLibrary

[soypat]

Looks like the default coloring for these characters before enabling semantic tokenizing is property.

[pjweinb]

This is more a vscode issue than a gopls issue, but here's the vscode story. Based on experiments I believe what happens is the following, to decide the foreground color:

1. If there is a semantic token, use that color.
2. If not, if there is a textmate label, use that color.
3. Otherwise, use the default color.

In this case the semantic token is a string, so vscode uses the color for strings. If gopls didn't return string semantic tokens, then vscode would show the colors from the textmate grammar, which would show different colors for \n etc. In semantic tokens I know of no way to get different colors in the string (i tried all 10 of the SemanticTokenModifiers, and they all use the same string color.)

So there are two possiblities for gopls:

1. Leave things the way they are. Then the vscode user's choice would be semantic tokens and drab strings, or turn off semantic tokens and get colorful strings.
2. Don't return string semantic tokens. Then vscode users would get colorful strings, but other LSP clients would get incomplete semantic tokens. I do not know if there are any LSP clients using semantic tokens, other than vscode.

[pjweinb]

See the comment in #45753.

There are only two choices, as I see it:

1. Leave things the way they are. Then the vscode user's choice would be semantic tokens and drab numbers, or turn off semantic tokens and get colorful numbers.
2. Don't return number semantic tokens. Then vscode users would get colorful numbers, but other LSP clients would get incomplete semantic tokens. I do not know if there are any LSP clients using semantic tokens, other than vscode.

[soypat]

See the comment in #45753.

There are only two choices, as I see it:

1. Leave things the way they are. Then the vscode user's choice would be semantic tokens and drab numbers, or turn off semantic tokens and get colorful numbers.

2. Don't return number semantic tokens. Then vscode users would get colorful numbers, but other LSP clients would get incomplete semantic tokens. I do not know if there are any LSP clients using semantic tokens, other than vscode.

@pjweinb I think this comment was maybe meant for #45792?

[pjweinb]

just substitute 'string' for 'number'. The choices are the same.

…

On Wed, Apr 28, 2021 at 4:26 PM Patricio Whittingslow < ***@***.***> wrote: See the comment in #45753 <#45753>. There are only two choices, as I see it: 1. Leave things the way they are. Then the vscode user's choice would be semantic tokens and drab numbers, or turn off semantic tokens and get colorful numbers. 2. Don't return number semantic tokens. Then vscode users would get colorful numbers, but other LSP clients would get incomplete semantic tokens. I do not know if there are any LSP clients using semantic tokens, other than vscode. I think this comment was maybe meant for #45792 <#45792>? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#45753 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJIAI5YSHRM74W7VUKZIUDTLBVP7ANCNFSM43QPKBLQ> .

[soypat]

Well I don't believe it is reasonable to be picky about what is implemented from the LSP. Option 1 seems like the better of two options. That said, is there any chance this could be brought in as an optional parameter to gopls? Something that can be modified in settings.json in VSCode for example.

[ian-h-chamberlain]

For an example of another LSP provider faced with a similar problem, see rust-lang/rust-analyzer#7111

Their solution was to simply add a config that enables/disables semantic tokens for strings only (as well as one for all semantic tokens), which I think seems like a good option in this case. Personally I'd love to have a way to disable it just for strings, since in my opinion the TM highlighting does look better than semantic tokens for strings (with escapes and format tokens), but isn't as good for the rest of the code.

I suppose a separate config for numbers or other literals might also make sense, in case a user wanted different behavior for different semantic types.

[pjweinb]

my plan is to add an option to turn semantic tokens off for both strings and numbers, but i could be persuaded to have two options. (this is all a mediocre idea; vscode could support better string and number semantic tokens)

…

On Tue, Aug 2, 2022 at 8:37 AM Ian Chamberlain ***@***.***> wrote: For an example of another LSP provider faced with a similar problem, see rust-lang/rust-analyzer#7111 <rust-lang/rust-analyzer#7111> Their solution was to simply add a config that enables/disables semantic tokens for strings only (as well as one for *all* semantic tokens), which I think seems like a good option in this case. Personally I'd love to have a way to disable it just for strings, since in my opinion the TM highlighting does look better than semantic tokens for strings (with escapes and format tokens), but isn't as good for the rest of the code. I suppose a separate config for numbers or other literals might also make sense, in case a user wanted different behavior for different semantic types. — Reply to this email directly, view it on GitHub <#45753 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJIAI2NLEMOIZWI2VTCPBTVXE6C7ANCNFSM43QPKBLQ> . You are receiving this because you were assigned.Message ID: ***@***.***>

[gopherbot]

Change https://go.dev/cl/421256 mentions this issue: internal/lsp: new options to disable certain kinds of semantic tokens

[usersina]

Change https://go.dev/cl/421256 mentions this issue: internal/lsp: new options to disable certain kinds of semantic tokens

"gopls": {
    "ui.semanticTokens": true,
    "ui.noSemanticString": true
 }