Support tmLanguage files natively

[chtenb]

I would like to make a feature request for support for tmLanguage files and/or the VSCode flavour of this. It is a very prevalent grammar format which would allow us to tap into a vastly bigger ecosystem directly. I'm aware that there are converters between the formats, however I've found that they don't work correctly in all cases. Moreover, it would save maintainers of grammars a lot of hassle to not have to maintain duplicate grammars formats for every single language.

[Enselic]

My personal opinion is that native support for .tmLanguage would be good to have in syntect. Especially if it can be achieved in a very modular way, essentially as a separate mod that can be turned off and on via a feature.

I do not find the idea of native .tmLanguage support very attractive if it is necessary to make broad, cross-cutting changes to the whole syntect code base. But I would guess such changes would not be necessary.

This is not the first time support for .tmLanguage has been discussed. For reference, here are some other instances where it was discussed or where native support would have helped:

• add syntax highlighting for Vala #67
• Curated list of languages #168
• SublimeText view.extract_scope feature #237
• does syntect support vscode tmlanguage files? #249

It would at least certainly have helped bat, because currently we need to do a lot of manual (albeit tool-assisted) conversions from .tmLanguage to .sublime-syntax when adding or upgrading syntaxes.

[schrieveslaach]

Yes, would be really nice, especially for bat (see sharkdp/bat#2129)

[michaelblyons]

Just an advisory for anyone thinking about this: Sublime does not support all tmLanguage tokens (I want to say \G notably?). So it may be more work for full support. And then on top of that, VSCode has things like injections.

[keith-hall]

Pretty sure ST has no problem with \G - the old Markdown syntax used to be littered with them. But yes, any tmLanguage features introduced in the last few years are likely unsupported by ST. It doesn't mean syntect can't support them though.

[yankeeinlondon]

Just wondering ... is this just a request or is there active development on this? Seems like a really powerful upgrade and am I right that support for Typescript is dependant on this?

[keith-hall]

No, Typescript support is dependant on #271 and #323

[acao]

VSCode has things like injections.

@michaelblyons indeed, as well as atom, and I think ace editor (that github uses). Github uses our grammars somehow. I would love to use this crate and dify or Pixelmatch to build a snapshot regression testing suite for our vscode-ified texmate grammars, but we make heavy use of injectionSelector, in some cases several layers deep (e.g. markdown -> python -> graphql). we might still use this just for the base grammar used for all graphql strings and files (discussion: graphql/graphiql#3273)

if anyone else would like to have advanced textmate features like injectionSelector support, I'd be happy to collaborate! I think it's possible that injectionSelector configs could be transformed into ST include configs at runtime if those are supported?