Skip to content

Commit

Permalink
feat(#1720): add instructions for running dbt locally
Browse files Browse the repository at this point in the history
  • Loading branch information
@witash
witash committed Nov 21, 2024
1 parent 50d0de2 commit 61cbd47
Show file tree
Hide file tree
Showing 3 changed files with 22 additions and 9 deletions.
5 changes: 4 additions & 1 deletion content/en/hosting/analytics/building-dbt-models.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,10 @@ packages:
```
To avoid breaking changes in downstream models, include `revision` in the dependency, which should be a version tag for `cht-pipeline`.

In the CHT Sync config, set the URL of dbt GitHub repository to the `CHT_PIPELINE_BRANCH_URL` [environment variable]({{< relref "hosting/analytics/environment-variables" >}}), either in `.env` if using Docker compose, or in `values.yaml` if using Kubernetes.
It is helpful to test changes locally before commiting them to a remote repository.
For local testing and devleopment, in the CHT Sync config, set the path to the dbt project to the `DBT_LOCAL_PATH` [environment variable]({{< relref "hosting/analytics/environment-variables" >}}) in `.env`
Use the [docker compose setup]({{< relref "hosting/analytics/setup-docker-compose" >}}) for CHT Sync and use `docker-compose.local.yml` instead of `docker-compose.yml`.
For production and remote deployments, in the CHT Sync config, set the URL of dbt GitHub repository to the `CHT_PIPELINE_BRANCH_URL` [environment variable]({{< relref "hosting/analytics/environment-variables" >}}), either in `.env` if using Docker compose, or in `values.yaml` if using Kubernetes.

{{% alert title="Note" %}}
If `CHT_PIPELINE_BRANCH_URL` is pointing to a private GitHub repository, you'll need an access token in the URL. Assuming your repository is `medic/cht-pipeline`, you would replace `<PAT>` with an access token: `https://<PAT>@github.com/medic/cht-pipeline.git#main`. Please see [GitHub's instructions](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) on how to generate a token. If you create a fine-grained access token you need to provide read and write access to the [contents](https://docs.github.com/en/rest/authentication/permissions-required-for-fine-grained-personal-access-tokens?apiVersion=2022-11-28#repository-permissions-for-contents) of the repository.
Expand Down
17 changes: 9 additions & 8 deletions content/en/hosting/analytics/environment-variables.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,14 @@ All the variables in the `.env` file:
| `POSTGRES_SCHEMA` | `v1` | PostgreSQL schema |
| `POSTGRES_TABLE` | `couchdb` | PostgreSQL table where the CouchDB data is copied |
| `POSTGRES_HOST` | `postgres` | PostgreSQL instance |
| `POSTGRES_PORT` | `5432` | PostgreSQL port |
| `CHT_PIPELINE_BRANCH_URL` | `"https://github.com/medic/cht-pipeline.git#main"` | cht-pipeline branch containing the dbt models |
| `POSTGRES_PORT` | `5432` | PostgreSQL port |
| `CHT_PIPELINE_BRANCH_URL` | `"https://github.com/medic/cht-pipeline.git#main"` | cht-pipeline branch containing the dbt models |
| `DBT_LOCAL_PATH` | `""` | if running dbt locally, path to the dbt project on the host |
| `DATAEMON_INTERVAL` | `5` | Interval (in minutes) for looking for new changes in the CouchDB data |
| `COUCHDB_USER` | `medic` | Username of the CouchDB instance |
| `COUCHDB_PASSWORD` | `password` | Password of the CouchDB instance |
| `COUCHDB_DBS` | `"medic"` | Comma separated list of databases to sync e.g `"medic, medic_sentinel"` |
| `COUCHDB_HOST` | `couchdb` | Host of the CouchDB instance |
| `COUCHDB_PORT` | `5984` | Port of the CouchDB instance |
| `COUCHDB_SECURE` | `false` | Is connection to CouchDB instance secure? |
| `COUCHDB_USER` | `medic` | Username of the CouchDB instance |
| `COUCHDB_PASSWORD` | `password` | Password of the CouchDB instance |
| `COUCHDB_DBS` | `"medic"` | Comma separated list of databases to sync e.g `"medic, medic_sentinel"` |
| `COUCHDB_HOST` | `couchdb` | Host of the CouchDB instance |
| `COUCHDB_PORT` | `5984` | Port of the CouchDB instance |
| `COUCHDB_SECURE` | `false` | Is connection to CouchDB instance use HTTPS? |

9 changes: 9 additions & 0 deletions content/en/hosting/analytics/setup-docker-compose.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,15 @@ For production CHT Core deployments, the port will most likely need to be set to
The first time you run the commands from any of the sections below it will need to download many Docker images and will take a while. You'll know it's done when you see `#8 DONE 0.0s` and you are returned to the command line. Be patient!
{{% /alert %}}

### Running DBT models locally

When developing DBT models, it is helpful to test changes locally before commiting them to a remote repository.
Set the path to the project to the `DBT_LOCAL_PATH` [environment variable]({{< relref "hosting/analytics/environment-variables" >}}) in `.env` and use `docker-compose.local.yml`.

```sh
docker compose -f docker-compose.postgres.yml -f docker-compose.local.yml up -d
```

### Separate CouchDB instance

This setup involves starting couch2pg, PostgreSQL, pgAdmin and dbt. It assumes you have a CouchDB instance running, and you updated the `.env` CouchDB variables accordingly.
Expand Down

0 comments on commit 61cbd47

Please sign in to comment.