Full jinja support in SQL files

[ramonvermeulen]

• Investigate the options to fully support dbt jinja in SQL files, ideally being able to click through {{ ref() }} and {{ source() }} tags to navigate through the dbt codebase.
• Implement a first working version that can be released, with at least support for most standard dbt jinja

[rinchinov]

Hi!

My name is Roman, I noticed this plugin it is looks like a huge work was done with lineage and js part. I am also developing a plugin for dbt https://github.com/rinchinov/ij-dbt-plugin it already can navigate, highlight macros and run models in few ways. I also thought about lineage and js part with navigation through DAG but JS is not my strong side and I bit stuck with it

I am wondering about cooperation between plugins. I belive with cooperation we can do JB ides more friendly to dbt projects. What do you think about this idea?

PS I am also was inspired by vs code plugin when start to create pluhins for IDEA

[ramonvermeulen]

Hi @rinchinov, nice to meet you!

Cool to see your dbt plugin, it seems to have some features that I have on the roadmap such as navigating through jinja macros which is pretty nice!

I actually started off trying some Java libraries for lineage rendering, but the end result wasn't so "visually pleasing" as I would like it to be. Then I decided to embed a CefBrowser with a react application in the plugin. It was primarily hard to find out how to make the "communication" bridge between the Kotlin and JavaScript runtime in the browser. I think as pro's it has that it looks a lot nicer, and you have a lot of freedom on what you want to do with the UI. Downside is that it is a little performance overhead, loading in a whole browser with react application within an IntelliJ plugin. Still looking for ways to improve the performance, one of them will be to not load the full lineage for large projects, but only the up/down stream of the selected model. For example I really notice it when loading up a dbt project with hundreds of models.

I think it would be cool to collaborate if that is possible, it also depends a bit on how well the two plugins (as in codebases) integrate I would say. I am not super happy with the code quality / state of the plugin right now, it is okay, but I think there is also a lot of improvement possible. One thing that I would also like to look into is validating the dbt manifest via the existing json schema's.

[rinchinov]

Hello @ramonvermeulen

I think we can investigate a way with extention points(https://plugins.jetbrains.com/docs/intellij/plugin-extension-points.html)

Yes, I used code generation from json schema to creare data classes serialization/deseralation for manifest, it works quite well(now I am updateting my plugin to the last version of ide, there is some issue with jackson lib and deserialization).

PS https://github.com/joshuataylor also interested in plugin. Maybe we can start small community around it.

PSS performance issues is pain for big DAGs, at my prod project were almost 2k of transformation, and default JB plugin for dbt frozes whole ide during parsing(there are need some optimisation anyway for big projects).

[ramonvermeulen]

Hello @ramonvermeulen

I think we can investigate a way with extention points(https://plugins.jetbrains.com/docs/intellij/plugin-extension-points.html)

Yes, I used code generation from json schema to creare data classes serialization/deseralation for manifest, it works quite well(now I am updateting my plugin to the last version of ide, there is some issue with jackson lib and deserialization).

PS https://github.com/joshuataylor also interested in plugin. Maybe we can start small community around it.

PSS performance issues is pain for big DAGs, at my prod project were almost 2k of transformation, and default JB plugin for dbt frozes whole ide during parsing(there are need some optimisation anyway for big projects).

Sounds good, or starting a new repo, taking best of both plugins and putting it into a new plugin might work as well? I'm down for starting a small community, and discussing the possibilities, maybe a slack/discord channel could be a good starting point?

[rinchinov]

Hey!

Sorry,
I was a bit busy last months. I think it is good staring point, lets create slack channel in dbt workspace maybe ?