Perf: Only recreate materialzed views when necessary

[bmtcril]

With the move of MVs to dbt, we currently rebuild every mv table on every dbt run / aspects init. This will be a problem with large datasets over time, we want to only rebuild those tables when absolutely necessary. dbt has the on_configuration_change configuration option to support this, we'll need to upstream it to dbt-clickhouse then update.

See: https://docs.getdbt.com/docs/build/materializations#materialized-view

[Ian2012]

@bmtcril are new MVs backfilled on creation?

[bmtcril]

@Ian2012 yes, and on every init / dbt run currently

[SoryRawyer]

After looking into this, the on_configuration_change setting is unlikely to prevent us from rebuilding the MVs on every run. As I understand it, there are two main approaches for us to avoid rebuilding MVs when we don't need to:

• The state:modified selector, which would require us to track the state of each dbt run over time, since that selector requires the previous runs' manifest to compare to.
• Adding a tag to each MV and selecting all models that don't have this tag. One approach could be to only select models materialized as views via the config selector