Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make the list-agg columns conditionally tested #516

Merged
merged 1 commit into from
Dec 5, 2024

Conversation

dave-connors-3
Copy link
Collaborator

This is a:

  • bug fix PR with no breaking changes
  • new functionality

Link to Issue

DBX tests are failing in main because there is no support on Databricks for an order_by clause in the dbt.listagg macro, so we can't guarantee the order of the array of values in models where we report a list of violations within a single record.

Example:

 dbt show -s fct_source_fanout -t databricks --output json     
14:25:37  Running with dbt=1.8.8
14:25:38  Found 70 models, 72 data tests, 28 seeds, 8 sources, 1 exposure, 2 metrics, 779 macros, 1 semantic model
14:25:41  {
  "node": "fct_source_fanout",
  "show": [
    {
      "parent": "source_1.table_2",
      "model_children": "stg_model_2, int_model_4"
    },
    {
      "parent": "source_1.table_1",
      "model_children": "stg_model_2, stg_model_1"
    }
  ]
}
(dpe) 
dbt-project-evaluator/integration_tests on  dbx-test-fixes via (dpe) on ☁️  dev-admin (us-east-1) on ☁️   took 5s 
❯ dbt show -s test_fct_source_fanout -t databricks --output json
14:25:47  Running with dbt=1.8.8
14:25:48  Found 70 models, 72 data tests, 28 seeds, 8 sources, 1 exposure, 2 metrics, 779 macros, 1 semantic model
14:25:55  {
  "node": "test_fct_source_fanout",
  "show": [
    {
      "parent": "source_1.table_2",
      "model_children": "int_model_4, stg_model_2"
    },
    {
      "parent": "source_1.table_1",
      "model_children": "stg_model_1, stg_model_2"
    }
  ]
}

You can see the values are reversed, but identical.

This PR makes those columns conditional in our test configuration so that they are tested for all other adapters except DBX.

Since CI is off, i am running these tests locally!

Integration Test Screenshot

image

Checklist

  • I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • BigQuery
    • Postgres
    • Redshift
    • Snowflake
    • Databricks
    • DuckDB
    • Trino/Starburst
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)

@dave-connors-3
Copy link
Collaborator Author

@b-per i did not test on trino bc i don't have it set up, and my BQ set up appears to be busted for some local issue -- i do not feel like this is a risky PR but lemme know if you want me to continue with manual tests

Copy link
Collaborator

@b-per b-per left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should maybe technically check on target.type in ['spark', 'databricks'] as people running vanilla Spark would have the same issue as the people running databricks.

But if you don't feel like we need to do this change, happy to merge as-is as well.

@dave-connors-3 dave-connors-3 merged commit c6d9c26 into main Dec 5, 2024
@dave-connors-3 dave-connors-3 deleted the dbx-test-fixes branch December 5, 2024 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants