Skip to content

Commit

Permalink
Add dbt-cloud integration command to dp cli
Browse files Browse the repository at this point in the history
# Documentation
  • Loading branch information
Radosław Dziadosz committed Apr 25, 2023
1 parent c5a151d commit a084a47
Show file tree
Hide file tree
Showing 3 changed files with 78 additions and 2 deletions.
2 changes: 1 addition & 1 deletion data_pipelines_cli/cli_commands/dbtcloud.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ def read_bigquery_config(env: str) -> Dict[str, Any]:
"--account_id",
type=int,
required=True,
help="""dbt Cloud Account identifier To obtain your dbt Cloud account ID, sign into dbt Cloud
help="""dbt Cloud Account identifier To obtain your dbt Cloud account ID, sign into dbt Cloud
in your browser. Take note of the number directly following the accounts path component of the
URL - this is your account ID""",
)
Expand Down
68 changes: 67 additions & 1 deletion docs/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -375,4 +375,70 @@ Example ``looker.yml`` file might look like this:
looker_project_id: my_looker_project
looker_webhook_secret: "{{ env_var('LOOKER_WEBHOOK_SECRET') }}"
looker_repository_branch: main
looker_instance_url: https://looker.company.com/
looker_instance_url: https://looker.company.com/
dbt Cloud configuration
++++++++++++++++++++++++++++++

``config/dbtcloud.yml`` contains configuration related to dbt Cloud:

.. list-table::
:widths: 25 20 55
:header-rows: 1

* - Parameter
- Data type
- Description
* - project_name
- string
- Name of the project to be created in dbt Cloud
* - schedule_interval
- string
- The cron expression with which the example job will be run
* - default_gcp_project
- string
- GCP project that will be used by default if a new environment is created in dbt Cloud
* - environments
- Array
- Details of the environments to be created in dbt Cloud

Configuration of the environments:

.. list-table::
:widths: 25 20 55
:header-rows: 1

* - Parameter
- Data type
- Description
* - name
- string
- Name of the environment that will be created in dbt Cloud
* - dataset
- string
- Target dataset for this environment
* - dbt_version
- string
- The dbt version used in this environment
* - bq_config_dir
- string
- The name of the dp env directory where the bigquery configuration for the environment is located. The name of the project in GCP will be read from it.

Example ``dbtcloud.yml`` file might look like this:

.. code-block:: yaml
project_name: "Data Pipelines Project"
schedule_interval: "0 12 * * *"
default_gcp_project: "default-project"
environments:
- name: "Develop"
dataset: "dev"
dbt_version: "1.0.0"
type: "development"
bq_config_dir: "dev"
- name: "Production"
dataset: "prod"
dbt_version: "1.0.0"
type: "deployment"
bq_config_dir: "prod"
10 changes: 10 additions & 0 deletions docs/integration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,13 @@ Looker

**dp** can generate lookML codes for your models and views, publish and deploy your `Looker <https://www.looker.com/>`_ project

dbt Cloud
++++++++++++++++++++++++++++++++++++++++++++++

The `Data Pipelines CLI` can configure a project in dbt Cloud. The following functions are supported:

- creation of a project
- adding a repository
- adding a connection to BigQuery
- creation of environments
- creation of sample jobs

0 comments on commit a084a47

Please sign in to comment.