Search highlight mostly hidden when semantic highlighting enabled

[Maska]

Describe the bug

When semantic highlighting is used, ST's search highlight is mostly hidden for hits within semantic tokens.

To Reproduce

Steps to reproduce the behavior:

1. Open Sublime Text in safe mode
2. Install Package Control
3. Upgrade Package Control to version 4
4. Install LSP
5. Install lsp-clangd (or some other LSP plugin capable of semantic highligting (haven't tested with others))
6. Configure semantic highlighting for LSP
7. In a file where LSP is active, search for strings that appear within semantic tokens

LSP.sublime-settings:

{
    "semantic_highlighting": true
}

The color for highlight is set so that hard-to-see highlights stand out better.
Mariana.sublime-color-scheme:

{
    "variables":
    {
    },
    "globals":
    {
        "highlight": "#00FF00"
    },
    "rules":
    [
        {
            "scope": "meta.semantic-token",
            "background": "#00000101"
        }
    ]
}

Example file with many sorts of occurrences of missing highlights when searching for "q":

void reproduce()
{
    // A semantic token gets a very faint highlight when searching.
    auto qqqq = "qqqqqqqq";

    // Error markings are drawn nicely.
    auto qqqqq qqqqqq;

    // Even-numbered rows.
    int qqqqqq;

    int qqqqqqq;

    // Odd-numbered rows. On some font sizes, the tops and bottoms are gone.
    int qqqqqqqq;

    int qqqqqqqqq;

    int i_hide_highlights_under_my_background;
    int qqqqqqqqqq;
    int i_hide_them_too;

    // qqqqqqqqqqqqq Comment highlights don't bother each other.
    // qqqqqqqqqqqqq But semantic tokens do bother comments.
    int qqqqqqqqqqq;
    // qqqqqqqqqqqqq

    // Highlight for selected text works.
    auto qq = 1;
    qq = 2;
    qq = 3;
    qq = 4;
}

Expected behavior

Search hits are highlighted very visibly, with surrounding, unbroken rounded rectangles, as shown in the image below.

In this screenshot, LSP's semantic highlighting is disabled.

Screenshots

Search highlights are buggy.

Highlights for selected text occurrences work fine.

The bug doesn't seem to depend on the theme used. Monokai and others suffer from it too.

With display scale 100%, the partial highlights disappear completely. Font size 48 is used to bring up details.

Environment

• OS: Windows 11 version 23H2 (OS Build 22631.3593)
• Display scale 150%
• Sublime Text version: 4180
• LSP version: 2.2.0
• Language servers used: Clangd 18.1.8, used through LSP-clangd 1.7.0

Additional context

This is likely related to #1934.

[rchl]

I haven't checked this locally but it looks like the upstream issue: sublimehq/sublime_text#817

It's also referenced in our list of blocking upstream issues: #2007

[Maska]

Thanks for the near-instant reply and for linking those resources. I'm not familiar with this project, so while searching with highlight-related terms, I didn't manage to find that list of upstream issues. This surely does seem to be one those.

[jwortmann]

I've tried to fix it via SessionView.initialize_region_keys, but it does not work; apparently the find region key (https://forum.sublimetext.com/t/what-is-the-find-results-highlight-key/69032/4) is automatically initialized by ST as soon as a new view is created. We would need to initialize the semantic highlighting keys before that happens, so I don't think this can be fixed on the LSP side.