Feature/add union data #133

fivetran-jamie · 2024-01-09T18:40:52Z

PR Overview

This PR will address the following Issue/Feature:
#130

This PR will result in the following new package version:

v0.16.0

Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:

includes source_relation in every:
- join
- window function partition
- generated surrogate key
- model uniqueness test
documents unioning ability in README
adds yml docs for source_relation
adds some missing documentation that was in the source README but not the transform
kinda by accident included changes from update hubspot_deal_changes description #132, which might be merged to main prior to this PR

dbt_hubspot v0.16.0

🎉 Feature Update 🎉

This release supports running the package on multiple Hubspot sources at once! See the README for details on how to leverage this feature (PR #133).

📝 Documentation 📝

Included documentation around dbt-core versions and disabling source freshness tests. Previously, we included this in the source package README but had missed the transform README (PR #133).
Tweaked the description of hubspot__deal_changes to better reflect the grain of the model (PR #132).

🛠️ Under the Hood 🛠️

Updated the maintainer PR template to resemble the most up to date format.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

dbt compile
dbt run –full-refresh
dbt run
dbt test
dbt run –vars (if applicable) different permutations of passthrough columns -- ran with diff hubspot__pass_through_all_columns values and specific passthrough columns with add_property_label: true

Before marking this PR as "ready for review" the following have been applied:

The appropriate issue has been linked and tagged
You are assigned to the corresponding issue and this PR
BuildKite integration tests are passing

Detailed Validation

Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":

You have validated these changes and assure this PR will address the respective Issue/Feature.
You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
You have provided details below around the validation steps performed to gain confidence in these changes.

See Hex notebook linked in Height

Standard Updates

Please acknowledge that your PR contains the following standard updates:

Package versioning has been appropriately indexed in the following locations:
- indexed within dbt_project.yml
- indexed within integration_tests/dbt_project.yml
CHANGELOG has individual entries for each respective change in this PR
README updates have been applied (if applicable)
DECISIONLOG updates have been updated (if applicable)
Appropriate yml documentation has been added (if applicable)

dbt Docs

Please acknowledge that after the above were all completed the below were applied to your branch:

docs were regenerated (unless this PR does not include any code or yml updates) - post approval

If you had to summarize this PR in an emoji, which would it be?

🍨

Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 29.0.4 to 41.0.0. - [Release notes](https://github.com/tj-actions/changed-files/releases) - [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md) - [Commits](tj-actions/changed-files@v29.0.4...v41.0.0) --- updated-dependencies: - dependency-name: tj-actions/changed-files dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]>

…ithub/workflows/tj-actions/changed-files-41.0.0 Bump tj-actions/changed-files from 29.0.4 to 41.0.0 in /.github/workflows

fivetran-joemarkiewicz

@fivetran-jamie the results of this PR look great! I just have a few questions and suggestions in the comments below that I would like your eyes on before we approve this. Let me know if you have any questions or want to chat about these further.

fivetran-joemarkiewicz · 2024-02-02T21:34:08Z

.github/workflows/check_docs.yml

@@ -18,7 +18,7 @@ jobs:

      - name: Get changed files
        id: changed-files
-        uses: tj-actions/changed-files@v29.0.4
+        uses: tj-actions/changed-files@v41.0.0


Let's remove this as we will use a different docs check this year

removed and added note about removal to the changelog

fivetran-joemarkiewicz · 2024-02-02T21:35:05Z

README.md

+
+  <details><summary><i>Expand for source configuration template</i></summary><p>
+
+> **Note**: If there are source tables you do not have (see [Step 4](https://github.com/fivetran/dbt_hubspot_source?tab=readme-ov-file#step-4-disable-models-for-non-existent-sources)), you may still include them here, as long as you have set the right variables to `False`. Otherwise, you may remove them from your source definition.


Same comment about if there is a more concise and manageable option for sharing this information with the users.

fivetran-joemarkiewicz · 2024-02-02T21:36:34Z

README.md

@@ -150,7 +1444,12 @@ vars:
  hubspot_service_enabled: true                           # Enables all service/ticket models. Default = false
  hubspot_ticket_deal_enabled: true                       # Default = false
 ```
+
+### Dbt-core Version Requirement for disabling freshness tests


We need to include the proper reference to dbt when mentioning it within our docs.

Suggested change

### Dbt-core Version Requirement for disabling freshness tests

### dbt Core™ Version Requirement for disabling freshness tests

removed it wholly

fivetran-joemarkiewicz · 2024-02-02T21:39:48Z

CHANGELOG.md

+# dbt_hubspot v0.16.0
+
+## 🎉 Feature Update 🎉 
+- This release supports running the package on multiple Hubspot sources at once! See the [README](https://github.com/fivetran/dbt_hubspot?tab=readme-ov-file#step-3-define-database-and-schema-variables) for details on how to leverage this feature ([PR #133](https://github.com/fivetran/dbt_hubspot/pull/133)).


I think it is also very relevant to include here that customers will not also see a new source_relation field in their end models.

Additionally, we should flag that customers using the ticket models will need to run a full refresh to capture the new schema change.

fivetran-joemarkiewicz · 2024-02-02T21:41:35Z

models/service/intermediate/int_hubspot__ticket_calendar_spine.sql

@@ -9,13 +9,13 @@
        file_format = 'parquet'
    )
 }}
-
+-- depends_on: {{ ref('stg_hubspot__ticket') }}


I worry this will now cause compilation errors for customers who try to run dbt compile when first trying to use the package.

What is the limitation of ensuring this model still references the source?

i'll add the same check i just added to zendesk so that non-unioning users will still use the source

fivetran-joemarkiewicz · 2024-02-02T21:41:59Z

packages.yml

+  # - package: fivetran/hubspot_source
+  #   version: [">=0.15.0", "<0.16.0"]
+  - git: https://github.com/fivetran/dbt_hubspot_source.git
+    revision: feature/add-union-data
+    warn-unpinned: false 


Reminder to update before merge.

fivetran-joemarkiewicz · 2024-02-02T22:00:39Z

README.md

@@ -150,7 +1444,12 @@ vars:
  hubspot_service_enabled: true                           # Enables all service/ticket models. Default = false
  hubspot_ticket_deal_enabled: true                       # Default = false
 ```
+
+### Dbt-core Version Requirement for disabling freshness tests
+If you are not using a source table that involves freshness tests, please be aware that the feature to disable freshness was only introduced in dbt-core 1.1.0. Therefore ensure the dbt version you're using is v1.1.0 or greater for this config to work.


Actually, I just realized this package requires the use of dbt v1.3.0 or higher. So I think we may be able to remove this messaging in both the source and transform.

dependabot bot and others added 6 commits January 2, 2024 16:46

update hubspot_deal_changes description

0c14102

pre-existing typo + add new description to model docs

dec16d0

first stab

67a5cb3

workign more

d6fb708

tests are passing and just added readme docs

fb4d04c

fivetran-jamie self-assigned this Jan 9, 2024

update pr template

9539248

fivetran-jamie marked this pull request as ready for review January 10, 2024 19:06

fivetran-jamie requested a review from fivetran-joemarkiewicz January 10, 2024 19:11

fivetran-jamie added 2 commits January 10, 2024 11:11

Merge pull request #131 from fivetran/dependabot/github_actions/dot-g…

af90a07

…ithub/workflows/tj-actions/changed-files-41.0.0 Bump tj-actions/changed-files from 29.0.4 to 41.0.0 in /.github/workflows

polish readme

64bdc1b

fivetran-joemarkiewicz requested changes Feb 2, 2024

View reviewed changes

fivetran-joemarkiewicz reviewed Feb 2, 2024

View reviewed changes

fivetran-jamie added 4 commits February 5, 2024 11:41

joe feedback

7cee0d4

swap source-definition template

f6ad5c7

another readme tweak

56df46c

add flags.WHICH conditional

d2ba15c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/add union data #133

Feature/add union data #133

fivetran-jamie commented Jan 9, 2024 •

edited

Loading

fivetran-joemarkiewicz left a comment

fivetran-joemarkiewicz Feb 2, 2024

fivetran-jamie Feb 5, 2024

fivetran-joemarkiewicz Feb 2, 2024

fivetran-jamie Feb 5, 2024

fivetran-joemarkiewicz Feb 2, 2024

fivetran-jamie Feb 5, 2024

fivetran-joemarkiewicz Feb 2, 2024

fivetran-jamie Feb 5, 2024

fivetran-joemarkiewicz Feb 2, 2024

fivetran-jamie Feb 5, 2024

fivetran-joemarkiewicz Feb 2, 2024

fivetran-joemarkiewicz Feb 2, 2024

fivetran-jamie Feb 5, 2024


		<details><summary><i>Expand for source configuration template</i></summary><p>

		> Note: If there are source tables you do not have (see [Step 4](https://github.com/fivetran/dbt_hubspot_source?tab=readme-ov-file#step-4-disable-models-for-non-existent-sources)), you may still include them here, as long as you have set the right variables to `False`. Otherwise, you may remove them from your source definition.

	### Dbt-core Version Requirement for disabling freshness tests
	### dbt Core™ Version Requirement for disabling freshness tests

Feature/add union data #133

Are you sure you want to change the base?

Feature/add union data #133

Conversation

fivetran-jamie commented Jan 9, 2024 • edited Loading

PR Overview

dbt_hubspot v0.16.0

🎉 Feature Update 🎉

📝 Documentation 📝

🛠️ Under the Hood 🛠️

PR Checklist

Basic Validation

Detailed Validation

Standard Updates

dbt Docs

If you had to summarize this PR in an emoji, which would it be?

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-jamie commented Jan 9, 2024 •

edited

Loading