Skip to content

Commit

Permalink
Merge branch 'current' into entra
Browse files Browse the repository at this point in the history
  • Loading branch information
matthewshaver authored Dec 4, 2024
2 parents 4dcd8e5 + 56d6e45 commit 63380c5
Show file tree
Hide file tree
Showing 11 changed files with 91 additions and 39 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ id: 2-how-we-style-our-sql
- ☁️ Use [SQLFluff](https://sqlfluff.com/) to maintain these style rules automatically.
- Customize `.sqlfluff` configuration files to your needs.
- Refer to our [SQLFluff config file](https://github.com/dbt-labs/jaffle-shop-template/blob/main/.sqlfluff) for the rules we use in our own projects.

- Exclude files and directories by using a standard `.sqlfluffignore` file. Learn more about the syntax in the [.sqlfluffignore syntax docs](https://docs.sqlfluff.com/en/stable/configuration.html#id2).
- Exclude files and directories by using a standard `.sqlfluffignore` file. Learn more about the syntax in the [.sqlfluffignore syntax docs](https://docs.sqlfluff.com/en/stable/configuration/index.html).
- Excluding unnecessary folders and files (such as `target/`, `dbt_packages/`, and `macros/`) can speed up linting, improve run times, and help you avoid irrelevant logs.
- 👻 Use Jinja comments (`{# #}`) for comments that should not be included in the compiled SQL.
- ⏭️ Use trailing commas.
- 4️⃣ Indents should be four spaces.
Expand Down
2 changes: 2 additions & 0 deletions website/docs/docs/build/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ This order is now in the "shipped" state, but we've lost the information about w

- To configure snapshots in versions 1.8 and earlier, refer to [Configure snapshots in versions 1.8 and earlier](#configure-snapshots-in-versions-18-and-earlier). These versions use an older syntax where snapshots are defined within a snapshot block in a `.sql` file, typically located in your `snapshots` directory.
- Note that defining multiple resources in a single file can significantly slow down parsing and compilation. For faster and more efficient management, consider the updated snapshot YAML syntax, [available in Versionless](/docs/dbt-versions/versionless-cloud) or [dbt Core v1.9 and later](/docs/dbt-versions/core).
- For more information on how to migrate from the legacy snapshot configurations to the updated snapshot YAML syntax, refer to [Snapshot configuration migration](/reference/snapshot-configs#snapshot-configuration-migration).


</VersionBlock>
Expand Down Expand Up @@ -496,6 +497,7 @@ To configure snapshots in versions 1.9 and later, refer to [Configuring snapshot
- In dbt versions 1.8 and earlier, snapshots are `select` statements, defined within a snapshot block in a `.sql` file (typically in your `snapshots` directory). You'll also need to configure your snapshot to tell dbt how to detect record changes.
- The earlier dbt versions use an older syntax that allows for defining multiple resources in a single file. This syntax can significantly slow down parsing and compilation.
- For faster and more efficient management, consider[ upgrading to Versionless](/docs/dbt-versions/versionless-cloud) or the [latest version of dbt Core](/docs/dbt-versions/core), which introduces an updated snapshot configuration syntax that optimizes performance.
- For more information on how to migrate from the legacy snapshot configurations to the updated snapshot YAML syntax, refer to [Snapshot configuration migration](/reference/snapshot-configs#snapshot-configuration-migration).

The following example shows how to configure a snapshot:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@ description: "Configure Snowflake connection."
sidebar_label: "Connect Snowflake"
---

:::note

dbt Cloud connections and credentials inherit the permissions of the accounts configured. You can customize roles and associated permissions in Snowflake to fit your company's requirements and fine-tune access to database objects in your account. See [Snowflake permissions](/reference/database-permissions/snowflake-permissions) for more information about customizing roles in Snowflake.

Refer to [Snowflake permissions](/reference/database-permissions/snowflake-permissions) for more information about customizing roles in Snowflake.

:::

The following fields are required when creating a Snowflake connection

| Field | Description | Examples |
Expand All @@ -14,9 +22,6 @@ The following fields are required when creating a Snowflake connection
| Database | The logical database to connect to and run queries against. | `analytics` |
| Warehouse | The virtual warehouse to use for running queries. | `transforming` |


**Note:** A crucial part of working with dbt atop Snowflake is ensuring that users (in development environments) and/or service accounts (in deployment to production environments) have the correct permissions to take actions on Snowflake! Here is documentation of some [example permissions to configure Snowflake access](/reference/database-permissions/snowflake-permissions).

## Authentication methods

This section describes the different authentication methods for connecting dbt Cloud to Snowflake. Configure Deployment environment (Production, Staging, General) credentials globally in the [**Connections**](/docs/deploy/deploy-environments#deployment-connection) area of **Account settings**. Individual users configure their development credentials in the [**Credentials**](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#get-started-with-the-cloud-ide) area of their user profile.
Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/collaborate/explore-projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,12 +164,12 @@ Under the the **Models** option, you can filter on model properties (access or m

<Expandable alt_header="Trust signals for resources" lifecycle="preview">

Trust signal icons offer a quick, at-a-glance view of data health when browsing your models in dbt Explorer. These icons keep you informed on the status of your model's health using the indicators **Healthy**, **Caution**, **Degraded**, and **Unknown**. For accurate health data, ensure the resource is up-to-date and has had a recent job run.
Trust signal icons offer a quick, at-a-glance view of data health when browsing your resources in dbt Explorer. These icons keep you informed on the status of your resource's health using the indicators **Healthy**, **Caution**, **Degraded**, and **Unknown**. For accurate health data, ensure the resource is up-to-date and has had a recent job run. Supported resources are models, sources, and exposures.

Each trust signal icon reflects key data health components, such as test success status, missing resource descriptions, absence of builds in 30-day windows, and more.

To access trust signals:
- Use the search function or click on **Models** or **Sources** under the **Resource** tab.
- Use the search function or click on **Models**, **Sources** or **Exposures** under the **Resource** tab.
- View the icons under the **Health** column.
- Hover over or click the trust signal to see detailed information.
- For sources, the trust signal also indicates the source freshness status.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,11 @@ Starting in Core 1.9, you can use the new [microbatch strategy](/docs/build/incr

Currently microbatch is supported on these adapters with more to come:
* postgres
* redshift
* snowflake
* bigquery
* spark
* databricks

### Snapshots improvements

Expand All @@ -68,6 +70,8 @@ Beginning in dbt Core 1.9, we've streamlined snapshot configuration and added a

Read more about [Snapshots meta fields](/docs/build/snapshots#snapshot-meta-fields).

To learn how to safely migrate existing snapshots, refer to [Snapshot configuration migration](/reference/snapshot-configs#snapshot-configuration-migration) for more information.

### `state:modified` improvements

We’ve made improvements to `state:modified` behaviors to help reduce the risk of false positives and negatives. Read more about [the `state:modified` behavior flag](#managing-changes-to-legacy-behaviors) that unlocks this improvement:
Expand Down
2 changes: 2 additions & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ Release notes are grouped by month for both multi-tenant and virtual private clo
\* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability.

## November 2024
- **Enhancement**: Trust signal icons in dbt Explorer are now available for Exposures, providing a quick view of data health while browsing resources. To view trust signal icons, go to dbt Explorer and click **Exposures** under the **Resource** tab. Refer to [Trust signal for resources](/docs/collaborate/explore-projects#trust-signals-for-resources) for more info.
- **Bug**: Identified and fixed an error with Semantic Layer queries that take longer than 10 minutes to complete.
- **Fix**: Job environment variable overrides in credentials are now respected for Exports. Previously, they were ignored.
- **Behavior change**: If you use a custom microbatch macro, set a [`require_batched_execution_for_custom_microbatch_strategy` behavior flag](/reference/global-configs/behavior-changes#custom-microbatch-strategy) in your `dbt_project.yml` to enable batched execution. If you don't have a custom microbatch macro, you don't need to set this flag as dbt will handle microbatching automatically for any model using the [microbatch strategy](/docs/build/incremental-microbatch#how-microbatch-compares-to-other-incremental-strategies).
- **Enhancement**: For users that have Advanced CI's [compare changes](/docs/deploy/advanced-ci#compare-changes) feature enabled, you can optimize performance when running comparisons by using custom dbt syntax to customize deferral usage, exclude specific large models (or groups of models with tags), and more. Refer to [Compare changes custom commands](/docs/deploy/job-commands#compare-changes-custom-commands) for examples of how to customize the comparison command.
Expand Down
20 changes: 20 additions & 0 deletions website/docs/guides/snowflake-qs.md
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,26 @@ Now that you have a repository configured, you can initialize your project and s
```
- In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message.
:::info
If you receive an insufficient privileges error on Snowflake at this point, it may be because your Snowflake role doesn't have permission to access the raw source data, to build target tables and views, or both.

To troubleshoot, use a role with sufficient privileges (like `ACCOUNTADMIN`) and run the following commands in Snowflake.

**Note**: Replace `snowflake_role_name` with the role you intend to use. If you launched dbt Cloud with Snowflake Partner Connect, use `pc_dbt_role` as the role.

```
grant all on database raw to role snowflake_role_name;
grant all on database analytics to role snowflake_role_name;

grant all on schema raw.jaffle_shop to role snowflake_role_name;
grant all on schema raw.stripe to role snowflake_role_name;

grant all on all tables in database raw to role snowflake_role_name;
grant all on future tables in database raw to role snowflake_role_name;
```
:::
## Build your first model
You have two options for working with files in the dbt Cloud IDE:
Expand Down
13 changes: 12 additions & 1 deletion website/docs/reference/commands/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,4 +83,15 @@ See [global configs](/reference/global-configs/print-output#print-color)

The `run` command supports the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly.

</VersionBlock>
</VersionBlock>

## Status codes

When calling the [list_runs api](/dbt-cloud/api-v2#/operations/List%20Runs), you will get a status code for each run returned. The available run status codes are as follows:

- Starting = 1
- Running = 3
- Success = 10
- Error = 20
- Canceled = 30
- Skipped = 40
63 changes: 33 additions & 30 deletions website/docs/reference/snapshot-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,30 +8,16 @@ meta:
import ConfigResource from '/snippets/_config-description-resource.md';
import ConfigGeneral from '/snippets/_config-description-general.md';


## Related documentation
* [Snapshots](/docs/build/snapshots)
* The `dbt snapshot` [command](/reference/commands/snapshot)

<!--
Parts of a snapshot:
- name
- query
-->

## Available configurations
### Snapshot-specific configurations

<ConfigResource meta={frontMatter.meta} />

<VersionBlock lastVersion="1.8">

import SnapshotYaml from '/snippets/_snapshot-yaml-spec.md';

<SnapshotYaml/>

</VersionBlock>

<Tabs
groupId="config-languages"
defaultValue="project-yaml"
Expand Down Expand Up @@ -78,8 +64,9 @@ snapshots:
[+](/reference/resource-configs/plus-prefix)[strategy](/reference/resource-configs/strategy): timestamp | check
[+](/reference/resource-configs/plus-prefix)[updated_at](/reference/resource-configs/updated_at): <column_name>
[+](/reference/resource-configs/plus-prefix)[check_cols](/reference/resource-configs/check_cols): [<column_name>] | all
[+](/reference/resource-configs/plus-prefix)[invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) : true | false
[+](/reference/resource-configs/plus-prefix)[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>}
[+](/reference/resource-configs/plus-prefix)[invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) : true | false
[+](/reference/resource-configs/plus-prefix)[dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current): <string>
```
</File>
Expand Down Expand Up @@ -112,8 +99,9 @@ snapshots:
[strategy](/reference/resource-configs/strategy): timestamp | check
[updated_at](/reference/resource-configs/updated_at): <column_name>
[check_cols](/reference/resource-configs/check_cols): [<column_name>] | all
[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>}
[invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) : true | false
[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>}
[dbt_valid_to_current](/reference/resource-configs/dbt_valid_to_current): <string>
```
</File>

Expand All @@ -123,11 +111,9 @@ snapshots:

<TabItem value="config-resource">

<VersionBlock firstVersion="1.9">
import LegacySnapshotConfig from '/snippets/_legacy-snapshot-config.md';

Configurations can be applied to snapshots using the [YAML syntax](/docs/build/snapshots), available in Versionless and dbt v1.9 and higher, in the `snapshot` directory file.

</VersionBlock>
<LegacySnapshotConfig />

<VersionBlock lastVersion="1.8">

Expand All @@ -150,11 +136,25 @@ Configurations can be applied to snapshots using the [YAML syntax](/docs/build/s

</Tabs>

### Snapshot configuration migration

The latest snapshot configurations introduced in dbt Core v1.9 (such as [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names), [`dbt_valid_to_current`](/reference/resource-configs/dbt_valid_to_current), and `hard_deletes`) are best suited for new snapshots. For existing snapshots, we recommend the following to avoid any inconsistencies in your snapshots:

#### For existing snapshots
- Migrate tables &mdash; Migrate the previous snapshot to the new table schema and values:
- Create a backup copy of your snapshots.
- Use `alter` statements as needed (or a script to apply `alter` statements) to ensure table consistency.
- New configurations &mdash; Convert the configs one at a time, testing as you go.

:::warning
If you use one of the latest configs, such as `dbt_valid_to_current`, without migrating your data, you may have mixed old and new data, leading to an incorrect downstream result.
:::

### General configurations

<ConfigGeneral />


<Tabs
groupId="config-languages"
defaultValue="project-yaml"
Expand All @@ -170,6 +170,7 @@ Configurations can be applied to snapshots using the [YAML syntax](/docs/build/s

<VersionBlock firstVersion="1.9">


```yaml
snapshots:
[<resource-path>](/reference/resource-configs/resource-path):
Expand Down Expand Up @@ -254,11 +255,8 @@ snapshots:

<TabItem value="config">

<VersionBlock firstVersion="1.9">
<LegacySnapshotConfig />

Configurations can be applied to snapshots using [YAML syntax](/docs/build/snapshots), available in Versionless and dbt v1.9 and higher, in the `snapshot` directory file.

</VersionBlock>

<VersionBlock lastVersion="1.8">

Expand Down Expand Up @@ -293,18 +291,23 @@ Snapshots can be configured in multiple ways:

<VersionBlock lastVersion="1.8">

1. Defined in YAML files using a `config` [resource property](/reference/model-properties), typically in your [snapshots directory](/reference/project-configs/snapshot-paths) (available in [Versionless](/docs/dbt-versions/versionless-cloud) or and dbt Core v1.9 and higher).
2. Using a `config` block within a snapshot defined in Jinja SQL
1. Defined in a YAML file using a `config` [resource property](/reference/model-properties), typically in your [snapshots directory](/reference/project-configs/snapshot-paths) (available in [Versionless](/docs/dbt-versions/versionless-cloud) or and dbt Core v1.9 and higher). The latest snapshot YAML syntax provides faster and more efficient management.
2. Using a `config` block within a snapshot defined in Jinja SQL.
3. From the `dbt_project.yml` file, under the `snapshots:` key. To apply a configuration to a snapshot, or directory of snapshots, define the resource path as nested dictionary keys.

Note that in Versionless and dbt v1.9 and later, snapshots are defined in an updated syntax using a YAML file within your `snapshots/` directory (as defined by the [`snapshot-paths` config](/reference/project-configs/snapshot-paths)). For faster and more efficient management, consider the updated snapshot YAML syntax, [available in Versionless](/docs/dbt-versions/versionless-cloud) or [dbt Core v1.9 and later](/docs/dbt-versions/core).

</VersionBlock>

Snapshot configurations are applied hierarchically in the order above with higher taking precedence.

### Examples
The following examples demonstrate how to configure snapshots using the `dbt_project.yml` file, a `config` block within a snapshot, and a `.yml` file.

<VersionBlock firstVersion="1.9">
The following examples demonstrate how to configure snapshots using the `dbt_project.yml` file and a `.yml` file.
</VersionBlock>

<VersionBlock lastVersion="1.8">
The following examples demonstrate how to configure snapshots using the `dbt_project.yml` file, a `config` block within a snapshot (legacy method), and a `.yml` file.
</VersionBlock>

- #### Apply configurations to all snapshots
To apply a configuration to all snapshots, including those in any installed [packages](/docs/build/packages), nest the configuration directly under the `snapshots` key:
Expand Down Expand Up @@ -397,7 +400,7 @@ The following examples demonstrate how to configure snapshots using the `dbt_pro

</File>

You can also define some common configs in a snapshot's `config` block. We don't recommend this for a snapshot's required configuration, however.
You can also define some common configs in a snapshot's `config` block. However, we don't recommend this for a snapshot's required configuration.

<File name='dbt_project.yml'>

Expand Down
4 changes: 4 additions & 0 deletions website/snippets/_legacy-snapshot-config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

:::info
Starting from Versionless and dbt Core v1.9, defining snapshots in a `.sql` file using a config block is a legacy method. You can define snapshots in YAML format using the latest [snapshot-specific configurations](/docs/build/snapshots#configuring-snapshots). For new snapshots, we recommend using these latest configs. If applying them to existing snapshots, you'll need to [migrate](#snapshot-configuration-migration) over.
:::
3 changes: 2 additions & 1 deletion website/src/components/expandable/styles.module.css
Original file line number Diff line number Diff line change
Expand Up @@ -145,4 +145,5 @@
.headerText {
display: flex;
align-items: center;
}
}

0 comments on commit 63380c5

Please sign in to comment.