Dbt glue with spark #120
Merged
Dbt glue with spark #120
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
p-pekala
added a commit
that referenced
this pull request
Dec 8, 2023
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <[email protected]> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Piotr Pękala <[email protected]> Co-authored-by: Pawel Pinkos <[email protected]> Co-authored-by: Pawel Jurkiewicz <[email protected]> Co-authored-by: pawelpinkos <[email protected]> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <[email protected]> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <[email protected]> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <[email protected]> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Piotr Pękala <[email protected]> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <[email protected]> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <[email protected]> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <[email protected]> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * Dbt glue with spark (#120) * Dbt glue with spark * Fixing dp cli * FIX #121 - Bump version and CHANGELOG for release 0.28.0 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Piotr Pękala <[email protected]> Co-authored-by: Pawel Pinkos <[email protected]> Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Pawel Jurkiewicz <[email protected]> Co-authored-by: pawelpinkos <[email protected]> Co-authored-by: github-actions <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <[email protected]> Co-authored-by: jszafran <[email protected]> Co-authored-by: michalrudko <[email protected]> Co-authored-by: Michal Rudko <[email protected]> Co-authored-by: ptutak-getindata <[email protected]>
p-pekala
added a commit
that referenced
this pull request
Dec 8, 2023
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <[email protected]> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Piotr Pękala <[email protected]> Co-authored-by: Pawel Pinkos <[email protected]> Co-authored-by: Pawel Jurkiewicz <[email protected]> Co-authored-by: pawelpinkos <[email protected]> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <[email protected]> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <[email protected]> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <[email protected]> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Piotr Pękala <[email protected]> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <[email protected]> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <[email protected]> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <[email protected]> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * Dbt glue with spark (#120) * Dbt glue with spark * Fixing dp cli * Dbt upgtade (#122) * Dbt upgrade to 1.7 * Dbt upgrade to 1.7 * Fixing dp cli * FIX #123 - Bump version and CHANGELOG for release 0.29.0 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Piotr Pękala <[email protected]> Co-authored-by: Pawel Pinkos <[email protected]> Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Pawel Jurkiewicz <[email protected]> Co-authored-by: pawelpinkos <[email protected]> Co-authored-by: github-actions <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <[email protected]> Co-authored-by: jszafran <[email protected]> Co-authored-by: michalrudko <[email protected]> Co-authored-by: Michal Rudko <[email protected]> Co-authored-by: ptutak-getindata <[email protected]>
p-pekala
added a commit
that referenced
this pull request
Dec 8, 2023
* Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * feat: looker integration for compile and deploy, new command: bi * tests fix, looker git with ssh * chore: organizing imports, removed not used variables * chore: dbt2looker as extra dependency, mypy errors fix, bi-git-key-path option instead of key-path * chore: formatting, types-requests dependency for mypy * refactor: refactoring bi, removing bi command, fix tests, moving bi flags to bi.yml instead of call option * Fix after rebase * Fix after rebase * Fix after rebase * Fix after rebase * Airbyte integration (#86) * [DATA-366] Update datahub package and add dbt extra (#79) Co-authored-by: Piotr Sierkin <[email protected]> * Documentation improvement (#80) * Added IAP Airbyte authorization with service account file Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Piotr Pękala <[email protected]> Co-authored-by: Pawel Pinkos <[email protected]> Co-authored-by: Pawel Jurkiewicz <[email protected]> Co-authored-by: pawelpinkos <[email protected]> * Service account OIDC ID token in deploy command (#91) * Added possibility to pass platform-agnostic service account OIDC ID token to deploy command * Added changelog information * added 'source freshness' command to dbt compile * Bump copier (#93) * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <[email protected]> * Documentation improvement (#80) * Documentation improvement * dp compile default environment has been set to local * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump datahub (#84) Co-authored-by: Piotr Sierkin <[email protected]> * [DATA-366] Update datahub package and add dbt extra (#79) * feat: update datahub package and add dbt extra * fix: replace datahub requirements * fix: pre-commit coma Co-authored-by: Piotr Sierkin <[email protected]> * Documentation improvement (#80) * Documentation improvement * GitPython is not required anymore (#83) Installation docs upgrade * chore: bump packages except copier * fix: package name * chore: add specific version for pyyaml * fix: misspelling * chore: test docs * chore: test * test * test * test * test * test * test * chore: bump packages * chore: add renovate * chore: bump copier * chore: update exception * fix: update by removing not existing flag * chore: update tests * feat: update schema on manifest to v7 * chore: bump packages * chore: delete renovate * chore: bump python version * chore: remove junk * chore: bring back old exception * chore: add matrix to tox * chore: add new require and classifier * trigger * chore(docs): update badge and changelog * chore(docs): replace old extension * chore: add description * chore(docs): add old suffix and envops * chore(docs): update changelog * chore(docs): update changelog * chore(docs): update copier * chore: pinup version for dbt2looker * chore(docs): update changelog * chore: replace var for more meaningful Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Piotr Pękala <[email protected]> * chore: downgrade version to properly prepare release * chore(docs): fix date in changelog * BI deployment is disabled when config is missing (#95) * DATA-629: Added Airbyte configuration documentation (#97) * feat: disable sync bucket (#102) * chore: add flag to turn off bucket syncing with artefacts * chore: replace default value * chore(docs): updat changelog * fix: add readthedocs config file * chore: repair failing tests * feat: add test for sync_bucket * chore(docs): cosmetic changes in changelog * chore: rename `sync-bucket` to `disable-bucket-sync` * chore(docs): update with flag * chore: replace `sync_bucket` with `disable_bucket_sync` * chore: update name for `_bucket_sync` --------- Co-authored-by: Piotr Sierkin <[email protected]> * Fix failing readthedocs builds (#104) * Removed sphinx action from prepare-release pipeline (#105) * Bump werkzeug from 2.2.2 to 2.2.3 (#106) * Bump werkzeug from 2.2.2 to 2.2.3 Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.2 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@2.2.2...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * bump codeclimate-action to 4.0.0 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pawel Jurkiewicz <[email protected]> * Airbyte fix creating multiple connections (#109) * Fixing tests. * Fix failing test. * Remove redundant line. * Add info about fix to CHANGELOG.md. * Add workspace_id entry to config. * Get rid of walrus operator & add test for missing workspace id. * Fix wrong connection id in Airflow ingestion task. * Fix typo in test name. * Add fetching default workspace if not available in config. * Increasign test coverage. * Update workspace_id configuration description. * Bumping Python version to 3.8 in prepare-release workflow. * Pin pydantic version (#112) * Pin Pydantic version until Copier fixes their dependency. * Add info to changelog. * Upgrade dbt to 1.5.4 (#114) * Upgrade dbt to 1.5.4 * Fixing imports for publish.py * Update of dbt_schema_version * Updating manifest.json for tests * Fix manifest for testing * Fixing manifest cntd. * Fixing manifest cntd. * Replace example-project with exampleproject * Adding tutorial for upgrading manifest in tests cntd. * Fixing typos in the docs * Version fixes before release --------- Co-authored-by: Michal Rudko <[email protected]> * Databricks generate job (#116) * new feature * small fixes * dbt databricks factory requirement * python 3.9 onwards * testing * linting * annotations fix * linting * no docs * linting * update docs * testing * tests with databricks * testenv * Upgrade pyyaml * Dbt glue (#118) * DBT glue * DBT glue * DBT glue * DBT glue * Dbt glue with spark (#120) * Dbt glue with spark * Fixing dp cli * Dbt upgtade (#122) * Dbt upgrade to 1.7 * Dbt upgrade to 1.7 * Fixing dp cli * Upgrade versions * Upgrade versions * FIX #124 - Bump version and CHANGELOG for release 0.30.0 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Piotr Pękala <[email protected]> Co-authored-by: Pawel Pinkos <[email protected]> Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Piotr Sierkin <[email protected]> Co-authored-by: Pawel Jurkiewicz <[email protected]> Co-authored-by: pawelpinkos <[email protected]> Co-authored-by: github-actions <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jakub Szafran <[email protected]> Co-authored-by: jszafran <[email protected]> Co-authored-by: michalrudko <[email protected]> Co-authored-by: Michal Rudko <[email protected]> Co-authored-by: ptutak-getindata <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
<description>Resolves
<issue nr here>Keep in mind: