Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve handling of overage spend #154

Merged
merged 3 commits into from
May 14, 2024
Merged

Improve handling of overage spend #154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cd43acb
Improve handling of overage spend
fernandobrito May 14, 2024
71705d1
Improve handling of overage spend
fernandobrito May 14, 2024
4f7e8b3
Prepare for release
fernandobrito May 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changes/5.1.1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
## dbt-snowflake-monitoring 5.1.1 - May 14, 2024

### Fixes

- Include overage spend on services recently introduced ([#154](https://github.com/get-select/dbt-snowflake-monitoring/pull/154))

### Contributors
- [@fernandobrito](https://github.com/fernandobrito) (Fixes)

10 changes: 10 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html),
and is generated by [Changie](https://github.com/miniscruff/changie).

## dbt-snowflake-monitoring 5.1.1 - May 14, 2024

### Fixes

- Include overage spend on services recently introduced ([#154](https://github.com/get-select/dbt-snowflake-monitoring/pull/154))

### Contributors
- [@fernandobrito](https://github.com/fernandobrito) (Fixes)


## dbt-snowflake-monitoring 5.1.0 - May 06, 2024

### Features
Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'dbt_snowflake_monitoring'
version: '5.1.0'
version: '5.1.1'
config-version: 2

profile: dbt_snowflake_monitoring
Expand Down
88 changes: 51 additions & 37 deletions models/hourly_spend.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@ hours as (
from hour_spine
),

-- GROUP BY to collapse possible overage and non-overage cost from the same service in the
-- same day into a single row so this model does not emit multiple rows for the same service
-- and hour
usage_in_currency_daily as (
select
usage_date,
account_locator,
replace(usage_type, 'overage-', '') as usage_type,
currency,
sum(usage_in_currency) as usage_in_currency,
from {{ ref('stg_usage_in_currency_daily') }}
group by all
),
ian-whitestone marked this conversation as resolved.
Show resolved Hide resolved

storage_terabytes_daily as (
select
date,
Expand Down Expand Up @@ -163,14 +177,14 @@ data_transfer_spend_hourly as (
null as storage_type,
null as warehouse_name,
null as database_name,
coalesce(stg_usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
coalesce(usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
spend as spend_net_cloud_services,
stg_usage_in_currency_daily.currency as currency
usage_in_currency_daily.currency as currency
from hours
left join {{ ref('stg_usage_in_currency_daily') }} as stg_usage_in_currency_daily on
stg_usage_in_currency_daily.account_locator = {{ account_locator() }}
and stg_usage_in_currency_daily.usage_type = 'data transfer'
and hours.hour::date = stg_usage_in_currency_daily.usage_date
left join usage_in_currency_daily on
usage_in_currency_daily.account_locator = {{ account_locator() }}
and usage_in_currency_daily.usage_type = 'data transfer'
and hours.hour::date = usage_in_currency_daily.usage_date
),

ai_services_spend_hourly as (
Expand All @@ -184,14 +198,14 @@ ai_services_spend_hourly as (
null as storage_type,
null as warehouse_name,
null as database_name,
coalesce(stg_usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
coalesce(usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
spend as spend_net_cloud_services,
stg_usage_in_currency_daily.currency as currency
usage_in_currency_daily.currency as currency
from hours
left join {{ ref('stg_usage_in_currency_daily') }} as stg_usage_in_currency_daily on
stg_usage_in_currency_daily.account_locator = {{ account_locator() }}
and stg_usage_in_currency_daily.usage_type = 'ai services'
and hours.hour::date = stg_usage_in_currency_daily.usage_date
left join usage_in_currency_daily on
usage_in_currency_daily.account_locator = {{ account_locator() }}
and usage_in_currency_daily.usage_type = 'ai services'
and hours.hour::date = usage_in_currency_daily.usage_date
),

logging_spend_hourly as (
Expand All @@ -204,14 +218,14 @@ logging_spend_hourly as (
null as storage_type,
null as warehouse_name,
null as database_name,
coalesce(stg_usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
coalesce(usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
spend as spend_net_cloud_services,
stg_usage_in_currency_daily.currency as currency
usage_in_currency_daily.currency as currency
from hours
left join {{ ref('stg_usage_in_currency_daily') }} as stg_usage_in_currency_daily on
stg_usage_in_currency_daily.account_locator = {{ account_locator() }}
and stg_usage_in_currency_daily.usage_type = 'logging'
and hours.hour::date = stg_usage_in_currency_daily.usage_date
left join usage_in_currency_daily on
usage_in_currency_daily.account_locator = {{ account_locator() }}
and usage_in_currency_daily.usage_type = 'logging'
and hours.hour::date = usage_in_currency_daily.usage_date
),

-- For now we just use the daily reported usage and evenly distribute it across the day
Expand All @@ -228,14 +242,14 @@ logging_spend_hourly as (
null as storage_type,
null as warehouse_name,
null as database_name,
coalesce(stg_usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
coalesce(usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
spend as spend_net_cloud_services,
stg_usage_in_currency_daily.currency as currency
usage_in_currency_daily.currency as currency
from hours
left join {{ ref('stg_usage_in_currency_daily') }} as stg_usage_in_currency_daily on
stg_usage_in_currency_daily.account_locator = {{ account_locator() }}
and stg_usage_in_currency_daily.usage_type = '{{ reader_usage_type }}'
and hours.hour::date = stg_usage_in_currency_daily.usage_date
left join usage_in_currency_daily on
usage_in_currency_daily.account_locator = {{ account_locator() }}
and usage_in_currency_daily.usage_type = '{{ reader_usage_type }}'
and hours.hour::date = usage_in_currency_daily.usage_date
),
{% endfor %}

Expand All @@ -246,14 +260,14 @@ reader_adj_for_incl_cloud_services_hourly as (
null as storage_type,
null as warehouse_name,
null as database_name,
coalesce(stg_usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
coalesce(usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
0 as spend_net_cloud_services,
stg_usage_in_currency_daily.currency as currency
usage_in_currency_daily.currency as currency
from hours
left join {{ ref('stg_usage_in_currency_daily') }} as stg_usage_in_currency_daily on
stg_usage_in_currency_daily.account_locator = {{ account_locator() }}
and stg_usage_in_currency_daily.usage_type = 'reader adj for incl cloud services'
and hours.hour::date = stg_usage_in_currency_daily.usage_date
left join usage_in_currency_daily on
usage_in_currency_daily.account_locator = {{ account_locator() }}
and usage_in_currency_daily.usage_type = 'reader adj for incl cloud services'
and hours.hour::date = usage_in_currency_daily.usage_date
),

reader_cloud_services_hourly as (
Expand All @@ -263,14 +277,14 @@ reader_cloud_services_hourly as (
null as storage_type,
null as warehouse_name,
null as database_name,
coalesce(stg_usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
coalesce(stg_usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) + reader_adj_for_incl_cloud_services_hourly.spend as spend_net_cloud_services,
stg_usage_in_currency_daily.currency as currency
coalesce(usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) as spend,
coalesce(usage_in_currency_daily.usage_in_currency / hours.hours_thus_far, 0) + reader_adj_for_incl_cloud_services_hourly.spend as spend_net_cloud_services,
usage_in_currency_daily.currency as currency
from hours
left join {{ ref('stg_usage_in_currency_daily') }} on
stg_usage_in_currency_daily.account_locator = {{ account_locator() }}
and stg_usage_in_currency_daily.usage_type = 'reader cloud services'
and hours.hour::date = stg_usage_in_currency_daily.usage_date
left join usage_in_currency_daily on
usage_in_currency_daily.account_locator = {{ account_locator() }}
and usage_in_currency_daily.usage_type = 'reader cloud services'
and hours.hour::date = usage_in_currency_daily.usage_date
left join reader_adj_for_incl_cloud_services_hourly on
hours.hour = reader_adj_for_incl_cloud_services_hourly.hour
),
Expand Down
Loading