Skip to content

Commit

Permalink
Merge branch 'current' into add-meta-tag
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Dec 11, 2024
2 parents 5617cf8 + f0d497b commit f0c9b4d
Show file tree
Hide file tree
Showing 10 changed files with 192 additions and 23 deletions.
15 changes: 3 additions & 12 deletions website/docs/docs/cloud/git/connect-gitlab.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Connecting your GitLab account to dbt Cloud provides convenience and another lay
- Clone repos using HTTPS rather than SSH.
- Carry GitLab user permissions through to dbt Cloud or dbt Cloud CLI's git actions.
- Trigger [Continuous integration](/docs/deploy/continuous-integration) builds when merge requests are opened in GitLab.
- GitLab automatically registers a webhook in your GitLab repository to enable seamless integration with dbt Cloud.

The steps to integrate GitLab in dbt Cloud depend on your plan. If you are on:
- the Developer or Team plan, read these [instructions](#for-dbt-cloud-developer-and-team-tiers).
Expand Down Expand Up @@ -114,20 +115,10 @@ If your GitLab account is not connected, you’ll see "No connected account". Se

Once you approve authorization, you will be redirected to dbt Cloud, and you should see your connected account. You're now ready to start developing in the dbt Cloud IDE or dbt Cloud CLI.


## Troubleshooting

### Errors when importing a repository on dbt Cloud project set up
If you do not see your repository listed, double-check that:
- Your repository is in a Gitlab group you have access to. dbt Cloud will not read repos associated with a user.

If you do see your repository listed, but are unable to import the repository successfully, double-check that:
- You are a maintainer of that repository. Only users with maintainer permissions can set up repository connections.

If you imported a repository using the dbt Cloud native integration with GitLab, you should be able to see the clone strategy is using a `deploy_token`. If it's relying on an SSH key, this means the repository was not set up using the native GitLab integration, but rather using the generic git clone option. The repository must be reconnected in order to get the benefits described above.

## FAQs

<FAQ path="Troubleshooting/gitlab-webhook"/>
<FAQ path="Troubleshooting/error-importing-repo"/>
<FAQ path="Git/gitignore"/>
<FAQ path="Git/gitlab-authentication"/>
<FAQ path="Git/gitlab-selfhosted"/>
Expand Down
1 change: 1 addition & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo
## December 2024

- **New**: [Dimensions](/reference/resource-configs/meta) now support the `meta` config property in [dbt Cloud "Latest" release track](/docs/dbt-versions/cloud-release-tracks) and from dbt Core 1.9. You can add metadata to your dimensions to provide additional context and information about the dimension. Refer to [meta](/reference/resource-configs/meta) for more information.
- **Fix**: [The dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) now respects the BigQuery [`execution_project` attribute](/docs/core/connect-data-platform/bigquery-setup#execution-project), including for exports.
- **New**: [Model notifications](/docs/deploy/model-notifications) are now generally available in dbt Cloud. These notifications alert model owners through email about any issues encountered by models and tests as soon as they occur while running a job.
- **New**: You can now use your [Azure OpenAI key](/docs/cloud/account-integrations?ai-integration=azure#ai-integrations) (available in beta) to use dbt Cloud features like [dbt Copilot](/docs/cloud/dbt-copilot) and [Ask dbt](/docs/cloud-integrations/snowflake-native-app) . Additionally, you can use your own [OpenAI API key](/docs/cloud/account-integrations?ai-integration=openai#ai-integrations) or use [dbt Labs-managed OpenAI](/docs/cloud/account-integrations?ai-integration=dbtlabs#ai-integrations) key. Refer to [AI integrations](/docs/cloud/account-integrations#ai-integrations) for more information.
- **New**: The [`hard_deletes`](/reference/resource-configs/hard-deletes) config gives you more control on how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. Note that `new_record` will create a new metadata column in the snapshot table.
Expand Down
3 changes: 3 additions & 0 deletions website/docs/docs/deploy/ci-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,8 @@ To validate _all_ semantic nodes in your project, add the following command to d

## Troubleshooting

<FAQ path="Troubleshooting/gitlab-webhook"/>

<DetailsToggle alt_header="Temporary schemas aren't dropping">
If your temporary schemas aren't dropping after a PR merges or closes, this typically indicates one of these issues:
- You have overridden the <code>generate_schema_name</code> macro and it isn't using <code>dbt_cloud_pr_</code> as the prefix.
Expand All @@ -201,6 +203,7 @@ A macro is creating a schema but there are no dbt models writing to that schema.

</DetailsToggle>


<DetailsToggle alt_header="Error messages that refer to schemas from previous PRs">

If you receive a schema-related error message referencing a <i>previous</i> PR, this is usually an indicator that you are not using a production job for your deferral and are instead using <i>self</i>. If the prior PR has already been merged, the prior PR's schema may have been dropped by the time the CI job for the current PR is kicked off.
Expand Down
14 changes: 14 additions & 0 deletions website/docs/faqs/Troubleshooting/error-importing-repo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
title: Errors importing a repository on dbt Cloud project set up
description: "Errors importing a repository on dbt Cloud project set up"
sidebar_label: 'Errors importing a repository on dbt Cloud project set up'
id: error-importing-repo
---

If you don't see your repository listed, double-check that:
- Your repository is in a Gitlab group you have access to. dbt Cloud will not read repos associated with a user.

If you do see your repository listed, but are unable to import the repository successfully, double-check that:
- You are a maintainer of that repository. Only users with maintainer permissions can set up repository connections.

If you imported a repository using the dbt Cloud native integration with GitLab, you should be able to see if the clone strategy is using a `deploy_token`. If it's relying on an SSH key, this means the repository was not set up using the native GitLab integration, but rather using the generic git clone option. The repository must be reconnected in order to get the benefits described above.
19 changes: 19 additions & 0 deletions website/docs/faqs/Troubleshooting/gitlab-webhook.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
title: Unable to trigger a CI job with GitLab
description: "Unable to trigger a CI job"
sidebar_label: 'Unable to trigger a CI job'
id: gitlab-webhook
---

When you connect dbt Cloud to a GitLab repository, GitLab automatically registers a webhook in the background, viewable under the repository settings. This webhook is also used to trigger [CI jobs](/docs/deploy/ci-jobs) when you push to the repository.

If you're unable to trigger a CI job, this usually indicates that the webhook registration is missing or incorrect.

To resolve this issue, navigate to the repository settings in GitLab and view the webhook registrations by navigating to GitLab --> **Settings** --> **Webhooks**.

Some things to check:

- The webhook registration is enabled in GitLab.
- The webhook registration is configured with the correct URL and secret.

If you're still experiencing this issue, reach out to the Support team at [email protected] and we'll be happy to help!
2 changes: 1 addition & 1 deletion website/docs/guides/mesh-qs.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ To set a production environment:
6. Click **Test Connection** to confirm the deployment connection.
6. Click **Save** to create a production environment.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/prod-settings.jpg" width="70%" title="Set your production environment as the default environment in your Environment Settings"/>
<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/prod-settings-1.png" width="100%" title="Set your production environment as the default environment in your Environment Settings"/>


## Set up a foundational project
Expand Down
15 changes: 10 additions & 5 deletions website/docs/reference/model-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,11 @@ models:
[+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): <materialization_name>
[+](/reference/resource-configs/plus-prefix)[sql_header](/reference/resource-configs/sql_header): <string>
[+](/reference/resource-configs/plus-prefix)[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail #only for materialized views on supported adapters
[+](/reference/resource-configs/plus-prefix)[unique_key](/reference/resource-configs/unique_key): <column_name_or_expression>

```


</File>

</TabItem>
Expand All @@ -57,6 +59,7 @@ models:
[materialized](/reference/resource-configs/materialized): <materialization_name>
[sql_header](/reference/resource-configs/sql_header): <string>
[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail #only for materialized views on supported adapters
[unique_key](/reference/resource-configs/unique_key): <column_name_or_expression>

```

Expand All @@ -69,12 +72,13 @@ models:

<File name='models/<model_name>.sql'>

```jinja
```sql

{{ config(
[materialized](/reference/resource-configs/materialized)="<materialization_name>",
[sql_header](/reference/resource-configs/sql_header)="<string>"
[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail #only for materialized views for supported adapters
[unique_key](/reference/resource-configs/unique_key)='column_name_or_expression'
) }}

```
Expand Down Expand Up @@ -212,7 +216,7 @@ models:
<VersionBlock lastVersion="1.8">
```jinja
```sql

{{ config(
[enabled](/reference/resource-configs/enabled)=true | false,
Expand All @@ -233,7 +237,7 @@ models:

<VersionBlock firstVersion="1.9">

```jinja
```sql

{{ config(
[enabled](/reference/resource-configs/enabled)=true | false,
Expand All @@ -246,8 +250,9 @@ models:
[persist_docs](/reference/resource-configs/persist_docs)={<dict>},
[meta](/reference/resource-configs/meta)={<dict>},
[grants](/reference/resource-configs/grants)={<dict>},
[contract](/reference/resource-configs/contract)={<dictionary>}
[event_time](/reference/resource-configs/event-time): my_time_field
[contract](/reference/resource-configs/contract)={<dictionary>},
[event_time](/reference/resource-configs/event-time)='my_time_field',

) }}

```
Expand Down
144 changes: 140 additions & 4 deletions website/docs/reference/resource-configs/unique_key.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,65 @@
---
resource_types: [snapshots]
resource_types: [snapshots, models]
description: "Learn more about unique_key configurations in dbt."
datatype: column_name_or_expression
---


<Tabs>

<TabItem value="models" label="Models">

Configure the `unique_key` in the `config` block of your [incremental model's](/docs/build/incremental-models) SQL file, in your `models/properties.yml` file, or in your `dbt_project.yml` file.

<File name='models/my_incremental_model.sql'>

```sql
{{
config(
materialized='incremental',
unique_key='id'
)
}}

```

</File>

<File name='models/properties.yml'>

```yaml
models:
- name: my_incremental_model
description: "An incremental model example with a unique key."
config:
materialized: incremental
unique_key: id

```

</File>

<File name='dbt_project.yml'>

```yaml
name: jaffle_shop

models:
jaffle_shop:
staging:
+unique_key: id
```
</File>
</TabItem>
<TabItem value="snapshots" label="Snapshots">
<VersionBlock firstVersion="1.9">
For [snapshots](/docs/build/snapshots), configure the `unique_key` in the your `snapshot/filename.yml` file or in your `dbt_project.yml` file.

<File name='snapshots/<filename>.yml'>

```yaml
Expand All @@ -23,6 +76,8 @@ snapshots:

<VersionBlock lastVersion="1.8">

Configure the `unique_key` in the `config` block of your snapshot SQL file or in your `dbt_project.yml` file.

import SnapshotYaml from '/snippets/_snapshot-yaml-spec.md';

<SnapshotYaml/>
Expand All @@ -49,10 +104,13 @@ snapshots:

</File>

</TabItem>
</Tabs>

## Description
A column name or expression that is unique for the inputs of a snapshot. dbt uses this to match records between a result set and an existing snapshot, so that changes can be captured correctly.
A column name or expression that is unique for the inputs of a snapshot or incremental model. dbt uses this to match records between a result set and an existing snapshot or incremental model, so that changes can be captured correctly.

In dbt Cloud "Latest" and dbt v1.9+, [snapshots](/docs/build/snapshots) are defined and configured in YAML files within your `snapshots/` directory. You can specify one or multiple `unique_key` values within your snapshot YAML file's `config` key.
In dbt Cloud "Latest" release track and from dbt v1.9, [snapshots](/docs/build/snapshots) are defined and configured in YAML files within your `snapshots/` directory. You can specify one or multiple `unique_key` values within your snapshot YAML file's `config` key.

:::caution

Expand All @@ -67,6 +125,32 @@ This is a **required parameter**. No default is provided.
## Examples
### Use an `id` column as a unique key

<Tabs>

<TabItem value="models" label="Models">

In this example, the `id` column is the unique key for an incremental model.

<File name='models/my_incremental_model.sql'>

```sql
{{
config(
materialized='incremental',
unique_key='id'
)
}}
select * from ..
```

</File>
</TabItem>

<TabItem value="snapshots" label="Snapshots">

In this example, the `id` column is used as a unique key for a snapshot.

<VersionBlock firstVersion="1.9">

<File name="snapshots/orders_snapshot.yml">
Expand Down Expand Up @@ -114,10 +198,38 @@ snapshots:

</File>

</TabItem>
</Tabs>

<VersionBlock firstVersion="1.9">

### Use multiple unique keys

<Tabs>
<TabItem value="models" label="Models">

Configure multiple unique keys for an incremental model as a string representing a single column or a list of single-quoted column names that can be used together, for example, `['col1', 'col2', …]`.

Columns must not contain null values, otherwise the incremental model will fail to match rows and generate duplicate rows. Refer to [Defining a unique key](/docs/build/incremental-models#defining-a-unique-key-optional) for more information.

<File name='models/my_incremental_model.sql'>

```sql
{{ config(
materialized='incremental',
unique_key=['order_id', 'location_id']
) }}
with...
```

</File>

</TabItem>

<TabItem value="snapshots" label="Snapshots">

You can configure snapshots to use multiple unique keys for `primary_key` columns.

<File name='snapshots/transaction_items_snapshot.yml'>
Expand All @@ -137,12 +249,35 @@ snapshots:
```

</File>
</TabItem>
</Tabs>
</VersionBlock>

<VersionBlock lastVersion="1.8">

### Use a combination of two columns as a unique key

<Tabs>
<TabItem value="models" label="Models">

<File name='models/my_incremental_model.sql'>

```sql
{{ config(
materialized='incremental',
unique_key=['order_id', 'location_id']
) }}
with...
```

</File>

</TabItem>

<TabItem value="snapshots" label="Snapshots">

This configuration accepts a valid column expression. As such, you can concatenate two columns together as a unique key if required. It's a good idea to use a separator (for example, `'-'`) to ensure uniqueness.

<File name='snapshots/transaction_items_snapshot.sql'>
Expand Down Expand Up @@ -170,7 +305,6 @@ from {{ source('erp', 'transactions') }}

Though, it's probably a better idea to construct this column in your query and use that as the `unique_key`:


<File name='models/transaction_items_ephemeral.sql'>

```sql
Expand Down Expand Up @@ -211,4 +345,6 @@ from {{ source('erp', 'transactions') }}
```

</File>
</TabItem>
</Tabs>
</VersionBlock>
2 changes: 1 addition & 1 deletion website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -941,6 +941,7 @@ const sidebarSettings = {
"reference/resource-configs/pre-hook-post-hook",
"reference/resource-configs/schema",
"reference/resource-configs/tags",
"reference/resource-configs/unique_key",
"reference/resource-configs/meta",
"reference/advanced-config-usage",
"reference/resource-configs/plus-prefix",
Expand Down Expand Up @@ -985,7 +986,6 @@ const sidebarSettings = {
"reference/resource-configs/strategy",
"reference/resource-configs/target_database",
"reference/resource-configs/target_schema",
"reference/resource-configs/unique_key",
"reference/resource-configs/updated_at",
],
},
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit f0c9b4d

Please sign in to comment.