[C Family] Full rewrite of C family syntaxes

[camila314]

I am currently working on a full rewrite of the C family using modern Sublime Text syntax highlighting features. I plan to rewrite C, Objective-C, C++, and Objective-C++ in that order. Note that as of right now, only the C syntax has been edited and the syntax test file does not yet have any annotations.

This is currently a draft. Please feel free to let me know if I'm missing anything and I will add it to the roadmap.

C Roadmap

• Numbers
• Strings
• Operators
• Punctuation
• Constants
• Control Flow (keywords, labels)
• Enclosings (parenthesis, brackets, braces)
• Default Types
• Storage modifiers and qualifiers
• Function calls
• Function definitions
• Structs, Enums, Unions, Typedefs
• Declspec and Attribute
• Standard C Functions
• Rewrite preprocessor
• Rewrite comments

Changes made to C Highlighting

• Removed #if 0 and #if 1 #else creating a commented out block
• Added i and j to numerical suffixes (GNU extension)
• Add _Alignof, alignof, and offsetof, _Static_assert, static_assert, _Pragma as word operators
• Every double-underscored identifier without another scope is treated as support constant
• __cplusplus is a support.constant
• Add nullptr_t (C23) and max_align_t (C11) to list of default types
• Add _Thread_local, thread_local, _Atomic, _Alignas and _Noreturn as storage modifiers
• Fixed bug where a function parameter followed by a newline wouldn't be highlighted properly
• Add more __declspec properties
• Add common __attribute__ properties
• Add digit separator ' (C23)
• Add scopes for __VA_ARGS__ and __VA_OPT__ (C23)
• Give defined macro a suport.macro.c scope
• Add #embed, #elifdef, and #elifndef directives (C23)
• Add multi-line double-slashed comments via \
• Add typedef support for function ptrs
• Function pointer arguments in a typedef now share the same scoping as regular function defs

Issues

• Fixes Incorrect syntax highlighting in C with GNU Statement Expressions #4058
• Fixes [C] Sublime doesn't index a function name on its own line preceded with an enumeration return type #4104
• Fixes [C] New C23 keywords #3956
• Fixes [C++] Go to function append the return type of the function to the function name if separated on a different line if the return type is a custom type #3938
• Fixes [C] Cannot goto functions defined after a very long function. #3901
• Fixes [C++] Shift/stream operators considered arithmetic operators #3805
• Fixes [C] Wrong scoping of number if sign and number are on different lines. #3592
• Fixes [C/C++] typedef marked const as an indexed entity.name.type #3504
• Fixes [C] pointer to function: return type is not highlighted #3317
• Fixes [C++] could not parse the function-call without left bracket in the same line #2815
• Fixes [C] Incorrect error indication when a function return type is on the line before the function name and struct/enum types are used #2762
• Fixes [C] Syntax highlighting around C namespaces is broken #2221
• Fixes [C] Bug with “register” keyword #2047
• Fixes [C] struct field is seen as a declaration inside a struct #1830
• Fixes [C/C++] The scope of function parameter types is wrong #1093
• Fixes [C/C++] Last function parameter highlighting breaks with newline before ending paren #1070
• [C] Add more scopes for variable assignment #3920
• [C++] Empty preprocessor line #3224
• [C] Macro struct converter #2334
• [Objective-C] Multi-parameter method names do not appear correctly in symbol list #1438
• [Objective-C] Incorrect syntax highlighting if newline after message target #1138

[camila314]

My motivations are simple

[jrappen]

heads up, you can test this in a test file for Sublime via:

// SYNTAX TEST "Packages/C/C.sublime-syntax" \

... \
//  ^ punctuation.separator.continuation \
...

[jrappen]

for line-continuation, change header to:

// SYNTAX TEST "Packages/C++/C.sublime-syntax" \

when testing:

... \
//  ^ punctuation.separator.continuation \
...

duplicate the line-separator for continuation to the line with the expected scopes as well.

[deathaxe]

Some general advices:

main and prototype are primary entry points and thus shouldn't be located somewhere in the middle of anywhere. They should be the first contexts.

I'd recommend to organize contexts in a logical tree-like top-down structure, grouping related things like preprocessors, declarations, statements, expressions, keywords, identifiers, constants, operators, variables etc. together.

Various syntaxes even use ###[ SECTION NAME]### comments to visually separate bigger blocks of related contexts. This also enables easy folding of irrelevant blocks.

All of that will certainly help to keep track of structure as syntax definition grows. Keeping same context order and structure especially is important when extending a syntax, which will be the casse for C++.

The more syntaxes in this repo follow a common schema the easier it is to maintain them.

Premature ordering can be kill of creativity, so order is not that important for prototyping. Maybe however don't wait too long to add some structure.

---

Please don't just add some code snippets to syntax test files, but also design assertions on the way going. It sometimes takes more time to write tests than syntax, they are however the only way to ensure not to break wanted/designed behavior by automated syntax tests.

When writing rules for certain constructs you have in mind what you find important. Tests are just an expression of those thoughts or tools to verify your ideas/expectations against reality and vice versa.

Tests can (or even should) also contain incomplete statements/expressions, to ensure code doesn't break too badly. At least it helps to ensure to limit breaking highlighting to small regions or makes you think about how to achieve it.

Assertions should also contain some negative tests to ensure meta scope boundaries are maintained correctly or to avoid unwanted overlapping meta scopes. These are the less obvious issues, which may however impact various functionalities such as syntax based folding.

[deathaxe]

FWIW, context order and structure of existing C syntaxes may not be ideal nor a good template to rely on. Better look at something more recent, such as Java, PHP or Python.

[michaelblyons]

I don't think those are the actual tests. 😉 I note the path to the syntax def in User, and the final scope being c-2.

May I recommend similar to your PackageDev clone, you look at the suggestions in the ReadMe of this repo? Then you can modify the C syntax directly. If you want to keep the old one for comparison, you can rename it or move that one to User.

I'm also curious why you changed the assertion comment from // to /* ... */. Edit: My mistake. I like line comments for syntax tests, but the old one is block comments.

[deathaxe]

C++ is expected to fail at this point. That's step 2. Maybe even disable its tests to be able to focus on C in the meanwhile.

[deathaxe]

Strings are commonly used in various contexts and thus may deserve a dedicated context.

Maybe something like:

Suggested change

	- match: \"
	scope: punctuation.definition.string.begin.c
	push: string.double
	- include: string.quoted.double

And in "literal" or "constants" section:

  string.quoted.double:
    - match: \"
      scope: punctuation.definition.string.begin.c
      push: string.quoted.double.body

  string.quoted.double.body:
    - match: \"
      scope: punctuation.definition.string.end.c
      pop: 1
  

[jrappen]

• You sometimes don't - include: immediately-pop the context you had already defined, but re-write the match. Same with - include: else-pop.
• you could make use of:

  variables:
    leading_wspace: (?:^\s*)

  as you're re-using that quite often.
• some scopes you use in ObjC have a C suffix
• word-list regexes should be moved to variables

[michaelblyons]

Some languages have extra subscopes for some of these.

[michaelblyons]

In no particular order, a bunch of this repository's open C Family issues: #409 #888 #1034 #1070 #1082 #1093 #1107 #1138 #1156 #1249 #1333 #1408 #1438 #1749 #1760 #1772 #1774 #1794 #1830 #1901 #1906 #1913 #1914 #1931 #1993 #2047 #2094 #2121 #2221 #2251 #2334 #2352 #2385 #2471 #2762 #2815 #2861 #2883 #2937 #3091 #3113 #3114 #3145 #3224 #3317 #3343 #3487 #3504 #3592 #3714 #3715 #3805 #3901 #3920 #3938 #3956 #3960 #3984 #4058 #4104

Not all of these may be valid, fixable, or even things we want. But it's at least worth looking at them to see if they happen to be fixed.

[uncenter]

I can't tell if these have been fixed yet, but some issues I noticed for C++ files on the latest stable that don't seem to be mentioned above:

The names of namespaces being used don't have any specific scopes relating them to namespaces.

I would expect the Typedef type here to have some scope related to type names.

[deathaxe]

This may easily cause owning contest to also break or not being terminated correctly due to brackets already being consumed here.

I'd recommend to use lookaheads and consume stray brackets only in top-level context.

[jrappen]

To auto-close related issues upon merge of this PR, GitHub expects certain keywords:

https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/using-keywords-in-issues-and-pull-requests

or in other words, you need to change the following in your opening comment (which will be the commit message of the merge commit):

## Issues fixed

- fixes #1070
- fixes #1093
...

[michaelblyons]

Edited OP to add GitHub's format.