Skip to content

Commit

Permalink
Merge pull request #368 from dbt-labs/tobie/primary_key_expansion
Browse files Browse the repository at this point in the history
Tobie/primary key expansion
  • Loading branch information
b-per authored Aug 9, 2023
2 parents 2180997 + 37690f8 commit 594f799
Show file tree
Hide file tree
Showing 6 changed files with 21 additions and 8 deletions.
4 changes: 4 additions & 0 deletions dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,10 @@ vars:

primary_key_test_macros: [["dbt.test_unique", "dbt.test_not_null"], ["dbt_utils.test_unique_combination_of_columns"]]

# -- Graph variables --
# node types to test for primary key coverage. acceptable node types: model, source, snapshot, seed
enforced_primary_key_node_types: ["model"]

# -- DAG variables --
models_fanout_threshold: 3

Expand Down
3 changes: 2 additions & 1 deletion docs/customization/overriding-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Currently, this package uses different variables to adapt the models to your obj
| `test_coverage_target` | the minimum acceptable test coverage percentage | 100% |
| `documentation_coverage_target` | the minimum acceptable documentation coverage percentage | 100% |
| `primary_key_test_macros` | the set(s) of dbt tests used to check validity of a primary key | `[["dbt.test_unique", "dbt.test_not_null"], ["dbt_utils.test_unique_combination_of_columns"]]` |
| `enforced_primary_key_node_types` | the set of node types for you you would like to enforce primary key test coverage. Valid options to include are `model`, `source`, `snapshot`, `seed` | `["model"]`

**Usage notes for `primary_key_test_macros:`**

Expand All @@ -21,7 +22,7 @@ For each entry in the parent list, the logic in `int_model_test_summary` will ev

Each set of test(s) that define a primary key requirement must be grouped together in a sub-list to ensure they are evaluated together (e.g. [`dbt.test_unique`, `dbt.test_not_null`] ).

*While it's not explicitly tested in this package, we strongly encourage adding a `not_null` test on each of the columns listed in the `dbt_utils.unique_combination_of_columns` tests.*
*While it's not explicitly tested in this package, we strongly encourage adding a `not_null` test on each of the columns listed in the `dbt_utils.unique_combination_of_columns` tests. Alternatively, on Snowflake, consider `dbt_constraints.test_primary_key` in the [dbt Constraints](https://github.com/Snowflake-Labs/dbt_constraints) package, which enforces each field in the primary key is non null.*

```yaml title="dbt_project.yml"
# set your test and doc coverage to 75% instead
Expand Down
6 changes: 6 additions & 0 deletions docs/rules/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ Apply a [uniqueness test](https://docs.getdbt.com/reference/resource-properties/
Additional tests can be configured by applying a [generic test](https://docs.getdbt.com/docs/building-a-dbt-project/tests#generic-tests) in the model's `.yml` entry or by creating a [singular test](https://docs.getdbt.com/docs/building-a-dbt-project/tests#singular-tests)
in the `tests` directory of you project.

**Enforcing on more node types(Advanced)**

You can optionally extend this test to apply to more node types (`source`,`snapshot`, `seed`). By configuring the variable `enforced_primary_key_node_types` to be a set of node types for which you wish to enforce primary key test coverage in addition to (or instead of) just models. Check out the [overriding variables section](../customization/overriding-variables.md) for instructions

Snapshots should always have a multi-field primary key in order to function, while sources and seeds may not. Depending on your expectations for duplicates and null values, different kinds of primary key tests may be appropriate. Consider your use case carefully.

---

## Test Coverage
Expand Down
11 changes: 5 additions & 6 deletions models/marts/tests/fct_missing_primary_key_tests.sql
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
{{
config(
alias = 'my_alias',
)
}}

with

tests as (
select * from {{ ref('int_model_test_summary') }}
where resource_type in
(
{% for resource_type in var('enforced_primary_key_node_types') %}'{{ resource_type }}'{% if not loop.last %},{% endif %}
{% endfor %}
)
),

final as (
Expand Down
1 change: 1 addition & 0 deletions models/marts/tests/fct_test_coverage.sql
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ with

test_counts as (
select * from {{ ref('int_model_test_summary') }}
where resource_type = 'model'
),

conversion as (
Expand Down
4 changes: 3 additions & 1 deletion models/marts/tests/intermediate/int_model_test_summary.sql
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,15 @@ agg_test_relationships as (
final as (
select
all_graph_resources.resource_name,
all_graph_resources.resource_type,
all_graph_resources.model_type,
coalesce(agg_test_relationships.is_primary_key_tested, FALSE) as is_primary_key_tested,
coalesce(agg_test_relationships.number_of_tests_on_model, 0) as number_of_tests_on_model
from all_graph_resources
left join agg_test_relationships
on all_graph_resources.resource_id = agg_test_relationships.direct_parent_id
where all_graph_resources.resource_type = 'model'
where
all_graph_resources.resource_type in ('model', 'seed', 'source', 'snapshot')
)

select * from final

0 comments on commit 594f799

Please sign in to comment.