Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Hourly Spellbook subproject #6401

Closed
wants to merge 13 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .github/workflows/commit_manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
runs-on: [ self-hosted, linux, spellbook-trino-ci ]
strategy:
matrix:
project: [ 'tokens', 'spellbook', 'daily_spellbook', 'nft', 'dex', 'solana']
project: [ 'tokens', 'spellbook', 'daily_spellbook', 'nft', 'dex', 'solana', 'hourly_spellbook']
max-parallel: 1

steps:
Expand Down Expand Up @@ -56,4 +56,4 @@ jobs:
run: "aws s3 cp $S3_LOCATION/manifest.json $S3_LOCATION/manifest_$GIT_SHA.json"

- name: upload manifest
run: "aws s3 cp $PROJECT_DIR/manifest.json $S3_LOCATION/manifest.json"
run: "aws s3 cp $PROJECT_DIR/manifest.json $S3_LOCATION/manifest.json"
20 changes: 20 additions & 0 deletions .github/workflows/hourly_spellbook.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: DBT Hourly Spellbook run

on:
workflow_dispatch:
pull_request:
paths:
- hourly_spellbook/**
- .github/workflows/hourly_spellbook.yml
- .github/workflows/dbt_run.yml

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
dbt-run:
uses: ./.github/workflows/dbt_run.yml
with:
project: 'hourly_spellbook'

4 changes: 2 additions & 2 deletions .github/workflows/prices_check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on:
workflow_dispatch:
pull_request:
paths:
- 'models/prices/*/*.sql'
- 'tokens/models/prices/*/*.sql'

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
Expand All @@ -30,7 +30,7 @@ jobs:
- name: Get prices diff
shell: bash
run: |
git diff ${{ github.event.pull_request.base.sha}}..${{ github.event.pull_request.head.sha }} models/prices/**/*.sql | grep '^\+ ' > scripts/new_lines.txt
git diff ${{ github.event.pull_request.base.sha}}..${{ github.event.pull_request.head.sha }} tokens/models/prices/**/*.sql | grep '^\+ ' > scripts/new_lines.txt

- name: Run validations
id: validations
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/spellbook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ on:
- '.gitignore'
- 'tokens/**'
- 'daily_spellbook/**'
- 'hourly_spellbook/**'
- 'nft/**'
- 'solana/**'

Expand Down
60 changes: 1 addition & 59 deletions dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,25 +65,6 @@ models:
dex:
+schema: dex

cex:
+schema: cex
arbitrum:
+schema: cex_arbitrum
avalanche_c:
+schema: cex_avalanche_c
bnb:
+schema: cex_bnb
ethereum:
+schema: cex_ethereum
optimism:
+schema: cex_optimism
polygon:
+schema: cex_polygon
fantom:
+schema: cex_fantom
bitcoin:
+schema: cex_bitcoin

balances:
+schema: balances
ethereum:
Expand Down Expand Up @@ -131,29 +112,6 @@ models:
ethereum:
+schema: sudoswap_ethereum

safe:
+schema: safe
ethereum:
+schema: safe_ethereum
optimism:
+schema: safe_optimism
gnosis:
+schema: safe_gnosis
polygon:
+schema: safe_polygon
bnb:
+schema: safe_bnb
avalanche_c:
+schema: safe_avalanche_c
fantom:
+schema: safe_fantom
arbitrum:
+schema: safe_arbitrum
goerli:
+schema: safe_goerli
base:
+schema: safe_base

aragon:
+schema: aragon
ethereum:
Expand Down Expand Up @@ -221,12 +179,7 @@ models:
ethereum:
+schema: fiat_dao_ethereum

cow_protocol:
+schema: cow_protocol
ethereum:
+schema: cow_protocol_ethereum
gnosis:
+schema: cow_protocol_gnosis


ironbank:
+schema: ironbank
Expand Down Expand Up @@ -335,17 +288,6 @@ models:
ethereum:
+schema: dydx_ethereum

lido:
+schema: lido
accounting:
+schema: lido_accounting
ethereum:
+schema: lido_accounting_ethereum
liquidity:
+schema: lido_liquidity
ethereum:
+schema: lido_liquidity_ethereum

layerzero:
+schema: layerzero
ethereum:
Expand Down
3 changes: 3 additions & 0 deletions hourly_spellbook/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Hourly Spellbook

This is a DBT sub project for the all hourly models in spellbook
100 changes: 100 additions & 0 deletions hourly_spellbook/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
#Welcome to the NFT Spellbook!

name: "hourly_spellbook"
version: "1.0.0"
config-version: 2

quoting:
database: false
schema: false
identifier: false

# This setting configures which "profile" dbt uses for this project.
# profile: "spellbook-poc-tokens"
profile: "spellbook-local"

vars:
DBT_ENV_CUSTOM_ENV_S3_BUCKET: "{{ env_var('DBT_ENV_CUSTOM_ENV_S3_BUCKET', 'local') }}"
DBT_ENV_INCREMENTAL_TIME: "{{ env_var('DBT_ENV_INCREMENTAL_TIME', '1') }}"
DBT_ENV_INCREMENTAL_TIME_UNIT: "{{ env_var('DBT_ENV_INCREMENTAL_TIME_UNIT', 'day') }}"
ETH_ERC20_ADDRESS: '0x0000000000000000000000000000000000000000'

# These configurations specify where dbt should look for different types of files.
# You don't need to change these!
model-paths: ["models", "../sources"]
# ../tests/* should be added to a separate shared folder
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["../macros", "macros"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
- "target"
- "dbt_packages"

# Configuring tests
# Full documentation: https://docs.getdbt.com/reference/test-configs
tests:
hourly_spellbook:
+store_failures: true # store failures for all tests

# Configuring seeds
# Full documentation: https://docs.getdbt.com/reference/seed-configs
# For configuring individual seeds (e.g. overriding column types) we recommend a yml file nested in the same folder as the seed
# example: seeds/cryptopunks/schema.yml
seeds:
hourly_spellbook:
+schema: test_data

# defining search order of macro invokation
dispatch:
- macro_namespace: dbt_utils
search_order: ['trino_utils', 'dbt_utils']

# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models
models:
hourly_spellbook:
+post-hook:
- sql: "{{ set_trino_session_property(is_materialized(model), 'writer_scaling_min_data_processed', model.config.get('writer_min_size', '500MB')) }}"
transaction: true
- sql: "{{ set_trino_session_property(is_materialized(model), 'task_scale_writers_enabled', false) }}"
transaction: true
- sql: "{{ optimize_spell(this, model.config.materialized) }}"
transaction: true
- sql: "{{ mark_as_spell(this, model.config.materialized) }}"
transaction: true
+materialized: view
+schema: no_schema # this should be overriden in model specific configs
+view_security: invoker

_project:
safe:
+schema: safe
ethereum:
+schema: safe_ethereum
optimism:
+schema: safe_optimism
gnosis:
+schema: safe_gnosis
polygon:
+schema: safe_polygon
bnb:
+schema: safe_bnb
avalanche_c:
+schema: safe_avalanche_c
fantom:
+schema: safe_fantom
arbitrum:
+schema: safe_arbitrum
goerli:
+schema: safe_goerli
base:
+schema: safe_base
cow_protocol:
+schema: cow_protocol
ethereum:
+schema: cow_protocol_ethereum
gnosis:
+schema: cow_protocol_gnosis

Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{{
config(
alias = 'tx_hash_labels_treasury_management_ethereum',

)
}}

Expand All @@ -10,7 +10,7 @@ with
-- DAO list taken from dao multisig list from Dune
-- https://github.com/duneanalytics/spellbook/blob/main/models/labels/dao/identifier/multisigs/labels_dao_multisig_ethereum.sql
SELECT distinct address
FROM {{ ref('labels_addresses') }}
FROM {{ source('labels','addresses') }}
WHERE category = 'dao' and blockchain = 'ethereum' and label_type = 'identifier'
),

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -209,104 +209,6 @@ models:
- name: evt_index
description: "event index of node record update"

- name: ens_ethereum_votes
meta:
blockchain: ethereum
sector: dao
contributors: soispoke
config:
tags: ['ethereum', 'votes', 'dao', 'cross-chain']
description: >
DAO votes on all chains across all contracts and versions
columns:
- name: blockchain
- name: project
description: "Project name of the DAO"
- name: version
description: "Version of the contract built and deployed by the DAO project"
- name: block_time
description: "UTC event block time of each DAO vote"
- name: tx_hash
description: "Unique transaction hash value tied to each vote on the DAO"
- &dao_name
name: dao_name
description: "DAO name"
- &dao_address
name: dao_address
description: "DAO wallet address"
- &proposal_id
name: proposal_id
- &votes
name: votes
description: "Votes weighted by the amount of governance tokens"
- &votes_share
name: votes_share
description: "Votes share in percent for a given proposal"
- &token_symbol
name: token_symbol
- &token_address
name: token_address
- &votes_value_usd
name: votes_value_usd
description: "USD amount of governance tokens used at the time of the vote"
- &voter_address
name: voter_address
- &support
name: support
description: "0 = Against, 1 = For, 2 = Abstain votes"
tests:
- accepted_values:
values: ['for', 'against', 'abstain']
- &reason
name: reason
description: "Optional onchain comments to explain votes"

- name: ens_ethereum_proposals
meta:
blockchain: ethereum
sector: dao
contributors: soispoke
config:
tags: ['ethereum', 'proposals', 'dao', 'cross-chain']
description: >
DAO proposals on all chains across all contracts and versions
columns:
- name: blockchain
- name: project
description: "Project name of the DAO"
- name: version
description: "Version of the contract built and deployed by the DAO project"
- name: created_at
description: "UTC event block time at which the proposal was created"
- name: tx_hash
description: "Unique transaction hash value tied to each DAO proposal"
- *dao_name
- *dao_address
- name: proposal_id
tests:
- unique
- &votes_for
name: votes_for
- &votes_against
name: votes_against
- &votes_total
name: votes_total
description: "Total number of governance tokens used to vote on a given DAO proposal"
- &number_of_voters
name: number_of_voters
- &participation
name: participation
description: "Participation in percent: Number of governance tokens used to vote / Total token supply"
- &status
name: status
description: "Proposal status: Queued, Active, Executed, Canceled or Defeated"
tests:
- accepted_values:
values: ['Queued', 'Active', 'Executed', 'Canceled','Defeated']
- &description
name: description
description: "Description of the proposal"

- name: ens_set_name_latest
meta:
blockchain: ethereum
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{{ config(
schema='lido_accounting_ethereum',
alias = 'buffer_inflow',

materialized = 'table',
file_format = 'delta',
post_hook='{{ expose_spells(\'["ethereum"]\',
Expand All @@ -21,7 +22,7 @@ union all
SELECT evt_block_time, amount, 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2, evt_tx_hash, date_trunc('day', evt_block_time) as day
FROM {{source('lido_ethereum','steth_evt_ELRewardsReceived')}}

union all
union all

SELECT evt_block_time, amount , 0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2, evt_tx_hash, date_trunc('day', evt_block_time) as day
FROM {{source('lido_ethereum','steth_evt_WithdrawalsReceived')}}
FROM {{source('lido_ethereum','steth_evt_WithdrawalsReceived')}}
Loading
Loading