Add retry mechanism for the given error

[hovaesco]

Describe the feature

Add retry mechanism for the given error, like for example DELTA_LAKE_BAD_WRITE.
That feature should be configurable, meaning that the user can specify one or more errors on which the query from the model will be retried. It should also include max number of retries and wait/sleep before the next attempt.

Describe alternatives you've considered

No response

Who will benefit?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[Thelin90]

In dbt we utilise something called artifacts:

• https://docs.getdbt.com/reference/artifacts/dbt-artifacts

We use this to store test results into a table called dbt_results , on our project level on-run-end.

Which outputs test results to a single table with all test results.

We don’t run all jobs in one single DBT job but they are spread out.

This leads to multiple DBT jobs finishing, trying to write on

on-run-end:
    - "{{ store_dbt_results(results) }}"

The delta write lock issue then kicks in here since multiple DBT jobs, within the same single cluster, try to write their output to the results table from the actual dbt run and dbt test.

So even if we spread out the run of the jobs, collision can happen. So this retry would be good when you have a few collisions, or rare ones, we can try to avoid it as much as possible due to how we run and schedule the DBT jobs, but we still need to be able to avoid collision, when having multiple run, test output tables in this case for each run.