fix: KeyError in CodeSyntaxHighlight

[hanjinliu]

I noticed that CodeSyntaxHighlight sometimes fails to highlight the texts. This PR fixes the KeyError just by replacing the __getitem__ with get. I encountered this bug when I tried to highlight my pre-commit-config.yaml file, so I cannot tell when this happens.

from superqt.utils import CodeSyntaxHighlight
from qtpy import QtWidgets as QtW

w = QtW.QPlainTextEdit()
highlight = CodeSyntaxHighlight(w.document(), "yaml", "default")
w.setPlainText("ci:\n  autoupdate_schedule: monthly")

Output:

KeyError                                  Traceback (most recent call last)
File ~\Desktop\code\python\superqt\src\superqt\utils\_code_syntax_highlight.py:73, in CodeSyntaxHighlight.highlightBlock(self, text)
     68 def highlightBlock(self, text):
     69     # dirty, dirty hack
     70     # The core problem is that pygemnts by default use string streams,
     71     # that will not handle QTextCharFormat, so we need use `data` property to
     72     # work around this.
---> 73     highlight(text, self.lexer, self.formatter)
     74     for i in range(len(text)):
     75         self.setFormat(i, 1, self.formatter.data[i])

File ~\mambaforge\envs\mt\Lib\site-packages\pygments\__init__.py:82, in highlight(code, lexer, formatter, outfile)
     77 def highlight(code, lexer, formatter, outfile=None):
     78     """
     79     This is the most high-level highlighting function. It combines `lex` and
     80     `format` in one function.
     81     """
---> 82     return format(lex(code, lexer), formatter, outfile)

File ~\mambaforge\envs\mt\Lib\site-packages\pygments\__init__.py:64, in format(tokens, formatter, outfile)
     62 if not outfile:
     63     realoutfile = getattr(formatter, 'encoding', None) and BytesIO() or StringIO()
---> 64     formatter.format(tokens, realoutfile)
     65     return realoutfile.getvalue()
     66 else:

File ~\Desktop\code\python\superqt\src\superqt\utils\_code_syntax_highlight.py:52, in QFormatter.format(self, tokensource, outfile)
     49 self.data = []
     51 for token, value in tokensource:
---> 52     self.data.extend([self._style[token]] * len(value))

KeyError: Token.Literal.Scalar.Plain

[Czaki]

For me, it looks like a bug in pigments.

[tlambert03]

For me, it looks like a bug in pigments.

what do you suggest then?

[Czaki]

I will check this in the following days. I am sorry that I forgot about this issue.

[Czaki]

So yes, it is error reported in pygments pygments/pygments#2149

Looking into rich code

https://github.com/Textualize/rich/blob/43d3b04725ab9731727fb1126e35980c62f32377/rich/syntax.py#L152

This PR may be a good approach. Hovewer I prefer to add Token.Literal.Scalar.Plain to self._style.

        self._style = {name: get_text_char_format(style) for name, style in self.style}

I feel like using get may mask some other problem. But rich author decided to mask (with adding to dict for performance), so maybe it is not a bad idea?

I cannot restart CI. Maybe too long time since last commit?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.98%. Comparing base (8a40170) to head (6728c02).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #258      +/-   ##
==========================================
- Coverage   87.13%   86.98%   -0.15%     
==========================================
  Files          46       46              
  Lines        3443     3443              
==========================================
- Hits         3000     2995       -5     
- Misses        443      448       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Czaki]

superqt still supports python 3.8 so need to use types from typing, not builtins

[tlambert03]

superqt still supports python 3.8

happy to drop it now that it's EOL. will open new issue

[tlambert03]

Hovewer I prefer to add Token.Literal.Scalar.Plain to self._style.

thanks @Czaki. is this something you want to see done in this PR before merge?