Backport Zig lexer from Notepad4 #267
Conversation
|
I was hesitating a bit whether Zig requires a special lexer and whether it wouldn't be enough to use e.g. the C lexer for it but there are the |
Should be no problem to add back if this is the preferred behavior for Scintilla. |
|
Zig documentation emphasizes that Doc Comments and Top Level Doc Comments are different so I expect users would want these differentiated. // Comments in Zig start with "//" and end at the next LF byte (end of line).
/// A structure for storing a timestamp, with nanosecond precision (this is a
/// multiline doc comment).
//! This module provides functions for retrieving the current date and
//! time with varying degrees of precision and accuracy. |
|
This character literal escape appears to lex incorrectly with the '9}' not in escape style. There doesn't appear to be a limit on number of digits for a '\u{1f4a9}'
|
set bool atEscapeEnd(int ch) noexcept {
--digitsLeft;
return (!brace && digitsLeft <= 0) || !IsHexDigit(ch);
} |
I added it back - unlike Dart, detection of function definitions is trivial (they always follow the
Done (based on @zufuliu's code).
Done. I used the lazy version |
|
Assuming that the Dart lexer gets merged first, I'll wait until it's merged and rebase this branch on top of it to resolve the conflicts. |
Just to be sure we don't wait for each other here - I more or less assumed that applies here too so I didn't perform the rebase. If something is needed from me for this PR, please let me know. |
|
@nyamatongwe Should I convert this one to the object lexer too? |
|
It would be best if the Zig lexer is also an object lexer. |
OK, I'll have a look at it. |
Done, please let me know if some changes are needed. I'm not sure whether 12, "SCE_ZIG_BUILTIN_FUNCTION", "identifier", "Builtin function",is the correct tag or some different tag should be used (builtin functions start with |
|
Builtin functions are mostly similar to other languages standard library functions so should allow applications to provide similar behaviour by specifying The use of syntax to distinguish builtins is an example where concepts don't cleanly match between languages. There could be additional tags to refine |
|
Most of this is working well but the folding isn't stable: it changes based on the ranges that are lexed. Open I may commit the lexer in this state, perhaps with folding turned off, to provide a basis for further work. This set of style definitions helps see the code for testing but its not good enough to distribute. |
Likely same bug as Bash comment line folding (issue #224), Dart lexer has same bug: Backtrack one line like Bash should fix the bug (backtracking is done inside Lines 1195 to 1201 in 75ae907 |
|
@zufuliu Thanks, it works! (To be clear, I don't know anything about how folding works and did just Ctrl+C Ctrl+V here.)
I'll try to reproduce it and if the bug is present, I'll submit the same patch for the Dart lexer too. |
|
Alright, I have one quite interesting observation - I first accidentally checked out the Dart lexer branch which wasn't converted to the object lexer and I couldn't reproduce the problem there. But once I realized it and switched to the branch using the object lexer, the problem appeared. Is there some difference between start line numbers between object/non-object lexers that could cause this? Isn't this a bug somewhere in Scintilla/Lexilla? |
|
I just added a Sci_Position lineCurrent = styler.GetLine(startPos);in Lex() both in the non-object and object lexer. Then I started SciTE moved caret down the screen and the first movement down that caused editor scroll was:
So something somewhere must cause this difference by 1 which I guess is not intended. |
|
For historical reasons, the adapter code |
Alright, so what should I do in the Zig and Dart case? Backtrack 1 line or to the beginning of the multiline string/comment? |
… document and lexer object from successful whole document fold for the per-line fold. Fixed by creating a new document and lexer for the per-line lexing and folding. This showed there are problems with the current Dart, F#, and Julia folders as well as the new Zig lexer in #267.
Start of string/comment. This should have been caught by the lexer testing program |
For Dart I did this: #275 If it's considered OK, I'll do the same for Zig here (plus I'll rebase this PR on top of master to get the changes in the test suite). |
|
The Dart change is OK although it could be shortened with |
This patch adds the Zig lexer from the Notepad4 editor originally created by Zufu Liu (@zufuliu). Some changes have been made to make it compile and work in Scintilla since Notepad4 contains a modified Scintilla version. Also, some features of the Notepad4 lexer have been removed/changed: - special highlighting of formatting strings has been removed as it uses some extra functions added to Notepad4's Scintilla and I didn't want to spend much time on figuring out what would have to be back-ported - other lexilla lexers don't do this either anyway - folding has been modified to use simple folding like the rest of Scintilla lexers and not folding of the previous line based on brace presence on the next line like Notepad4 - "semi-syntactic" coloring of Notepad4 which colors functions following "fn" has been removed as this is not performed in other Scintilla lexers - highlighting of tasks such as TODOs in comments has been removed as it isn't present in other Scintilla lexers - colorig states and keywords have been renamed slightly - keywords at index 3 have been reserved for caller-supplied type names using SCI_SETKEYWORDS like in the C lexer (which we use in Geany to supply type names obtained using ctags) Fixes ScintillaOrg#237.
I.e. identifiers after the "fn" keywords.
Done.
Done for Zig. |
This patch adds the Zig lexer from the Notepad4 editor originally created by Zufu Liu (@zufuliu) and convert it to an object lexer. Some changes have been made to make it compile and work in Scintilla since Notepad4 contains a modified Scintilla version. Also, some features of the Notepad4 lexer have been removed/changed: - special highlighting of formatting strings has been removed as it uses some extra functions added to Notepad4's Scintilla and I didn't want to spend much time on figuring out what would have to be back-ported - other lexilla lexers don't do this either anyway - folding has been modified to use simple folding like the rest of Scintilla lexers and not folding of the previous line based on brace presence on the next line like Notepad4 - "semi-syntactic" coloring of Notepad4 which colors functions following "fn" has been removed as this is not performed in other Scintilla lexers - highlighting of tasks such as TODOs in comments has been removed as it isn't present in other Scintilla lexers - colorig states and keywords have been renamed slightly - keywords at index 3 have been reserved for caller-supplied type names using SCI_SETKEYWORDS like in the C lexer (which we use in Geany to supply type names obtained using ctags) Add SCE_ZIG_COMMENTLINETOP state for top-level comments. Add styling function definitions I.e. identifiers after the "fn" keywords. Fixes #237.
|
Committed with some minor edits. The ID number clashed with |
This patch adds the Zig lexer from the Notepad4 editor originally created by Zufu Liu (@zufuliu).
Some changes have been made to make it compile and work in Scintilla since Notepad4 contains a modified Scintilla version. Also, some features of the Notepad4 lexer have been removed/changed:
Full diff is here:
Full diff
Note: Fear not, this is the last lexer I'm planning to steal from Notepad4 :-).
Fixes #237.