chore: Merge workflows, test on all available targets #400
Conversation
erezrokah
force-pushed
the
chore/better_tests
branch
5 times, most recently
from
c060e28 to
1490bda
Compare
| @@ -1,5 +1,5 @@ | |||
| {% macro is_snowflake() %} | |||
| {% if target.name == 'snowflake' %} | |||
There was a problem hiding this comment.
I don't think this worked at our target name in the profiles.yml is dev-snowflake.
I believe the Snowflake tests were passing due to the --select argument added in https://github.com/cloudquery/policies-premium/pull/166/files#diff-5529d7636471ea70d566f8a3ea390efc26e7d7f7df536b3c65f7066567112757R54 and b9f2782#diff-5529d7636471ea70d566f8a3ea390efc26e7d7f7df536b3c65f7066567112757R54
There was a problem hiding this comment.
So Snowflake tests were not really running
| script: | | ||
| const fs = require('fs/promises'); | ||
| const { TRANSFORMATION_DIR: transformation_dir } = process.env; | ||
| const [postgres, snowflake, bigquery] = await Promise.all([ |
There was a problem hiding this comment.
The main change in this PR is to check for the relevant DB target specs and run migrate + DBT on each existing target. So you can add a new target to the CI by simply adding a bigquery.yml file for example
| @@ -1,3 +1,5 @@ | |||
| {{ config(enabled=is_postgres()) }} | |||
There was a problem hiding this comment.
I tried to merge the README notes on what is supported per target plugin the select argument from https://github.com/cloudquery/policies-premium/pull/166/files#diff-5529d7636471ea70d566f8a3ea390efc26e7d7f7df536b3c65f7066567112757R54 and b9f2782#diff-5529d7636471ea70d566f8a3ea390efc26e7d7f7df536b3c65f7066567112757R54 when adding is_postgres or is_snowflake.
Please verify these are correct (the AWS compliance premium policy is missing the supported models, it only lists the free ones)
erezrokah
force-pushed
the
chore/better_tests
branch
from
c8ccece to
4f852a7
Compare
Still work in progress, merged the
_postgresand_snowflakeworkflows and also add tests for BigQuery.The workflows will check for relevant specs under
testsand will run:tests/postgres.ymlexiststests/snowflake.ymlexiststests/bigquery.ymlexistsI'll create a matching
tests/bigquery.ymlfile for GCP compliance premium based on https://github.com/cloudquery/policies-premium/blob/f2fdc9dd7e2574a2ba6f3b5f27a1f6c362307154/transformations/gcp/compliance-premium/tests/profiles.yml#L24 just from our integration testing account sincecq-playgroundis nuked on a schedule