My motivations are simple

Some general advices:

main and prototype are primary entry points and thus shouldn't be located somewhere in the middle of anywhere. They should be the first contexts.

I'd recommend to organize contexts in a logical tree-like top-down structure, grouping related things like preprocessors, declarations, statements, expressions, keywords, identifiers, constants, operators, variables etc. together.

Various syntaxes even use ###[ SECTION NAME]### comments to visually separate bigger blocks of related contexts. This also enables easy folding of irrelevant blocks.

All of that will certainly help to keep track of structure as syntax definition grows. Keeping same context order and structure especially is important when extending a syntax, which will be the casse for C++.

The more syntaxes in this repo follow a common schema the easier it is to maintain them.

Premature ordering can be kill of creativity, so order is not that important for prototyping. Maybe however don't wait too long to add some structure.

Please don't just add some code snippets to syntax test files, but also design assertions on the way going. It sometimes takes more time to write tests than syntax, they are however the only way to ensure not to break wanted/designed behavior by automated syntax tests.

When writing rules for certain constructs you have in mind what you find important. Tests are just an expression of those thoughts or tools to verify your ideas/expectations against reality and vice versa.

Tests can (or even should) also contain incomplete statements/expressions, to ensure code doesn't break too badly. At least it helps to ensure to limit breaking highlighting to small regions or makes you think about how to achieve it.

Assertions should also contain some negative tests to ensure meta scope boundaries are maintained correctly or to avoid unwanted overlapping meta scopes. These are the less obvious issues, which may however impact various functionalities such as syntax based folding.

FWIW, context order and structure of existing C syntaxes may not be ideal nor a good template to rely on. Better look at something more recent, such as Java, PHP or Python.

• update commit message for merge (i.e. opening comment)
• tests for comments should include - comment comment tests if they cannot be nested
• add tests for indentation and (transformed) symbols (you need to switch to "dev" in the top right when viewing the docs page)
• maybe you could add ###[ SECTION HEADERS ]##################### to C++, too
• you sometimes use invalid.illegal.xxx.yyy.zzz, sometimes invalid.illegal.xxx-yyy-zzz
• you've got a lot of capturing groups where you do not use them, that decreases performance
• you could add a syntax var to C, to use in the first_line_matches
• move word lists from matches to variables
• syntax variables should never include capturing groups
• looking at the syntax test files, it seems you copied & pasted a lot.
  • Tests for the base syntax should cover (1) its base scope and (2) whatever you have defined there.
  • Tests for the extending syntaxes should cover (3) their base scopes and (4) whatever is different in their syntax compared to whatever syntax(es !) they are extending. There is no need for and you should not include duplicate tests.

First of all let me say I'm happy someone is looking at these syntaxes. I'm impressed by the sheer speed you are displaying.

I haven't read the 50+ posts on this pull request but I'm trying it out locally now, specifically the C++ syntax.

Problem 1: Eager scoping of entity.name.function

.

The first thing I'm noticing is that for plain function calls, it assigns entity.name.function. Regrettably, I would want this behavior as it was before. Function calls should be scoped variable.function. Examples where it too eagerly assigns entity.name.function:

C& C::f(std::string_view z) noexcept
{
    if (const auto e = a().b(z)) {
        return f(*e); // f *should* be scoped as variable.function
    }
    return *this;
}

C& C::f(const X& x) noexcept
{
    const std::vector<X*> xs = f().g<X>(); // the call to g<X> *should* be variable.function
    m = std::distance(xs.begin(),
        std::find_if(xs.begin(), xs.end(), [&](const X* c) -> bool {
            return c == std::addressof(x);
        })
    );
    return *this;
}

void Frame::OnOpenProject(wxCommandEvent& /*event*/)
{
    if (/*...current content has not been saved...*/ 0) {
        const wxString body("Current content has not been saved! Proceed?");
        //             ^^^^ variable.function
        const wxString title("Please confirm");
        //             ^^^^^ variable.function
        if (wxMessageBox(body, title, wxICON_QUESTION | wxYES_NO, this) == wxNO)
            return;
    }
    wxFileDialog openFileDialog(this, wxT("Open a project file"), "", "", "JSON files (*.json)|*.json", wxFD_OPEN);
    //           ^^^^^^^^^^^^^^ variable.function
    if (openFileDialog.ShowModal() == wxID_CANCEL)
        return;
    try {
        Project project(Project::LoadFromFile(openFileDialog.GetPath()));
        //      ^^^^^^^ variable.function
        SetCurrentWorkingDirectory(project.GetBaseDir());
        // ^^^^^^^^^^^^^^^^^^^^^^^ variable.function
    } catch (const std::exception& ex) {
        wxLogError("Failed to load project: %s", ex.what());
        // ^^^^^^^ variable.function
        SetStatusText("project NOT loaded");
        // ^^^^^^^^^^ variable.function
        return;
    }
    SetStatusText("project loaded!");
}

This eager scoping of entity.name.function also happens in class initializer lists:

SomeClass::SomeClass(const S& title, const P& pos, const Z& size)
    : m_member(std::make_unique<G>(m_other_member))
    //              ^^^^^^^^^^^ variable.function
{
}

It also too eagerly assigns entity.name.function to macro invocations:

void C::Update()
{
    LOG_INFO(FACILITY) << "hello there general kenobi";
    // ^^^^^ - entity.name.function
    //                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ string.quoted.double
    const auto& state = getState();
    if (m.Update(state)) {
        LOG_INFO(FACILITY) << "frobnicating";
        // ^^^^^ - entity.name.function
        //                    ^^^^^^^^^^^^^^ string.quoted.double
        getFoo().f(getBlorb());
        LOG_INFO(FACILITY) << "did the frobnicating";
        // ^^^^^ - entity.name.function
        //                    ^^^^^^^^^^^^^^^^^^^^^^ string.quoted.double
    }
}

Problem 2: Does not handle templates well

The following are all valid C++ template classes:

template <class T> struct X {
//               ^ punctuation.definition.generic.end
};

template <class T, class S> class Y {
//                        ^ punctuation.definition.generic.end
};

template <typename T, typename S> class Z {
//                              ^ punctuation.definition.generic.end    
};

template <template <class> T> struct W {
//                       ^ punctuation.definition.generic.end
//                          ^ punctuation.definition.generic.end
};

Compiler Explorer link: https://godbolt.org/z/o5P3MrjGM

Template functions are also not scoped correctly:

#include <type_traits>
#include <utility>

template <typename T> struct is_pyobject : std::false_type {};
//                  ^ punctuation.definition.generic.end

struct object {};

template <typename T, std::enable_if_t<is_pyobject<T>::value, int> = 0>
//                                                                    ^ punctuation.definition.generic.end
auto object_or_cast(T &&o) -> decltype(std::forward<T>(o)) {
    return std::forward<T>(o);
}

template <typename T, std::enable_if_t<!is_pyobject<T>::value, int> = 0>
//                                                                     ^ punctuation.definition.generic.end

object object_or_cast(T &&o);
//                          ^ - invalid.illegal

int main() {
    return 0;
}

Compiler Explorer link: https://godbolt.org/z/xb7zKb85j

For working with templates, I invite you to clone this project locally and see if you can make the template-heavy code in the header files work: https://github.com/pybind/pybind11/tree/master/include/pybind11.

I'm just reporting global themed problems as I go :) thanks once again for tackling this!