Plugins
A set of common tools for the plugins is located in Tools/misc.py.
I've found that Python files in subdirectories like this can't be reloaded while Sublime Text is running, and it must be restarted for any changes to take effect.
That's a real pain for development so hopefully it's not necessary to change that file too often, or I'll just move it back to the top directory.
The highlighter plugins are derived from the BaseHighlighter class, which handles two main jobs:
- Searching the file for regions which match the selector scope names (given by the derived class) and marking them up if they pass the required tests (defined by the derived class), and then managing those regions by deleting them later if the text is modified.
- Handling the logic of running the highlighter tests.
By direct analogy to, and very much inspiration from, the existing test suite for ST syntaxes, the behaviour of each highlighter is verified by a set of highlighter tests.
These are located in the Tools/ directory with the standard syntax tests, because it's nice to keep the top level directory free of clutter.
The idea of the highlighter tests is to walk through an example file that's been marked up by the highlighter in question, and check that the internal name of each region (which is decided by the derived highlighter) matches the name that it is expected to be.
The expectation of what the region should be named is given on an assertion line (comment line starting with #!), and sets of characters on the assertion line called pointers will literally point at the characters around them which are expected to be drawn with a region of the asserted name.
Highlighter tests follow a very similar pattern to standard syntax tests, so if you're familiar with those then you should just get it right away. But there are a few improvements that I made, based on my own prior frustrations with the standard syntax tests:
- Each assertion line may have multiple discontiguous sets of pointers (
<-,^, etc) separated by any amount of whitespace, as long as they all assert the same region name, which is determined to start at the first character that isn't a pointer and extend to the end of the line. - Two extra types of pointers exist:
<--for pointing at the character second in from the start of the line (whereas<-identifies the character first in from the start) andVfor pointing at the character on the next line down. A consequence of the letter "V" being used as a pointer means that no region name can start with a capital V or it won't be interpreted correctly. The way to make this easy is to simply say that region names should never contain any capital letters; they're just not allowed.
If one of ^ or V is used on the line, then that must be the only type of up/down pointer on the line, but left pointers are allowed in either case.
The acceptable formats of an assertion line are as follows:
#![<- ][<-- ][^ [^ ]...](name)
#![<- ][<-- ][V [V ]...](name)
where any whitespace may be entirely excluded or indefinitely repeated, [] marks an optional element, ... indicates repetition of the previous optional element, and () marks a requisite element.
There is an additional improvement that I'm only able to make because of the fact this problem is a little simplified with respect to the standard syntax tests:
-
The absence of any assertion pointing at a given character in the test file is itself an assertion: it's an implicit assertion that character will not have a region drawn on it. Consequently, the highlighter tests don't need to understand the
- <name>style of negated assertion from the standard syntax tests, because we achieve the same simply by not having any explicit assertion at all.This comes in handy to make sure that no extra regions are being drawn in the test file than we expect, without any need for explicitly asserting that certain characters won't have some particular named region. You will see this a lot in the highlighter tests; sections of fish script that are only there to make sure the highlighter doesn't mark them up in any way.
The results pane for the highlighter tests is just the same as that of the standard syntax tests, and you can interact with it in the same way. Despite some small grievances, it's really a good system.
The compatibility highlighter is actually predominantly programmed by an external file, highlighter_compat_rules.yaml.
This is in YAML because it's easier to handle as a developer and evokes the style of .sublime-syntax files, even though it does add an external dependency (luckily Package Control can take care of it).
Mostly the rules file exists to define the scope selectors to look for as potential issues, and then which version of fish to be targeting to consider them definitely an issue. The complete file format (where ... indicates allowed repetition of the above structure) is as follows:
changes:
<change type>: <problem text>
...
issues:
<issue name>:
selector: <scope OR list of scopes>
match: <regex string OR boolean true>
history:
- change: <change type>
version: <fish version number>
...
hint: <hint text>
...Each issue must be identified by a unique issue name. It must have one or more scope selectors (selector) used to pick candidate structures from the source file. To have multiple scope selectors, a Python-style list is used.
The full text in the source file with the selected scope is now a candidate issue, and is tested against the given regular expression (match). Alternatively, if the expression is "true" then no testing is done and the candidate is immediately accepted. (This is much more efficient, because we can have all the regex matching done instead during syntax parsing. To this end, regex testing may be removed entirely in future.)
If the candidate is accepted, then the fish version being targeted by the plugin (see Compatibility highlighter) is compared to each individual change of state throughout the history of the issue. If a conflict is determined (eg, the plugin is targeting fish 2.7 but the issue history indicates the structure was only added in fish 3.0) then the full extent of the structure will be highlighted.
For the user, hovering over the region will reveal the problem that describes what's going wrong, and the hint that suggests alternative code to use to resolve the conflict.
The problem is rendered to the user according to the table at the top of the file, which maps the internal change type to a user-friendly string (which will be Python format()'ed to insert the version number).
Currently it isn't possible for two regions to overlap on the same character, because frankly I think this looks bad, and extra work would be needed to decide which popup to show on hover. I believe it could also lead to users being confused about which problem to try and solve first, particularly when the issues are related (as they most likely are, if they're trying to highlight the exact same characters).
For this reason, the order that issues appear in the list dictates the priority for showing issues which can apply to the same regions. Once a region has been drawn in a file, no other issue will be allowed to intersect with that region, and if one would then it is not drawn. In general, for cases where there are multiple issues that could coincide on the same regions, the issue which is most "serious" should must appear before the others in the list so that it will be drawn preferentially.
