Merge pull request #16 from stellar/add-tags-to-models

Add tags to models

models/docs/marts/ledger_current_state/account_signers_current.md

@@ -0,0 +1,11 @@
+[comment]: < Account Signers Current -
+
+{% docs account_signers_current %}
+
+The `account_signers_current` table is a nightly snapshotted table that represents the current status of all account signers associated with an account. The table returns the latest account entry, which is defined as the highest `last_modified_ledger` per account signer on the Stellar Network. Deleted signers are included in the table. For full state history, please use the `account_signers` table.
+
+**The `account_signers_current` table is only updated nightly. Intraday ledger state changes are not captured in the table.**
+
+The `account_signers_current` table may be joined to the `accounts` table in order to find out more information about the originating account. The signers table has a many-to-one relationship with the accounts table, so you should expect multiple records to be returned for each `account_id`.
+
+{% enddocs %}

models/docs/marts/ledger_current_state/accounts_current.md

@@ -0,0 +1,11 @@
+[comment]: < Accounts Current -
+
+{% docs accounts_current %}
+
+The `accounts_current` table is a nightly snapshotted table that represents the current state of account ledger entries. The table returns the latest account entry, which is defined as the highest `last_modified_ledger` per account on the Stellar Network. Deleted accounts are included in the table. For full state history, please use the `accounts` table.
+
+**The `accounts_current` table is only updated nightly. Intraday ledger state changes are not captured in the table.**
+
+As a reminder, account ledger entries store detailed information for a given account, including current account status, preconditions for transaction authorization, security settings and account balance. The balance reported in the accounts table reflects the account’s XLM balance only. All other asset balances are reported in the `trust_lines` table.
+
+{% enddocs %}

models/docs/marts/ledger_current_state/liquidity_pools_current.md

@@ -0,0 +1,15 @@
+[comment]: < Liquidity Pools Current -
+
+{% docs liquidity_pools_current %}
+
+The `liquidity_pools_current` table is a nightly snapshotted table that represents the current status of all liquidity pools. The table returns the latest pool entry, which is defined as the highest `last_modified_ledger` per liquidity pool on the Stellar Network. Deleted pools are included in the table. For full state history, please use the `liquidity_pools` table.
+
+**The `liquidity_pools_current` table is only updated nightly. Intraday ledger state changes are not captured in the table.**
+
+{% enddocs %}
+
+{% docs asset_pair %}
+
+A concatenated representation of the pair of assets in the liquidity pool, in the form `asset_a_code:asset_b_code`. In the case when the pool contains `XLM`, `XLM` is written as an asset code even though the native asset normally has a null asset code. Asset pair intends to make a human readable name for a pool, but does **not** represent a unique pool id. Different asset issuers can mint asset codes of the same name. In these cases, there will be duplicate `asset_pair` names. The pool ids are unique.
+
+{% enddocs %}

models/docs/marts/ledger_current_state/offers_current.md

@@ -0,0 +1,11 @@
+[comment]: < Offers Current -
+
+{% docs offers_current %}
+
+The `offers_current` table is a nightly snapshotted table that represents the current orderbook of the Stellar Decentralized Exchange. The table returns the latest offer, which is defined as the highest `last_modified_ledger` per offer on the Stellar Network. Deleted offers are included in the table. An offer is deleted when the seller either updates their offer price to zero or an order is completely filled. For full orderbook history, please use the `offers` table.
+
+**The `offers_current` table is only updated nightly. Intraday ledger state changes are not captured in the table.**
+
+For more information on how the Stellar Decentralized Exchange works, please read [these docs](https://developers.stellar.org/docs/encyclopedia/liquidity-on-stellar-sdex-liquidity-pools#order-books).
+
+{% enddocs %}

models/docs/marts/ledger_current_state/trust_lines_current.md

@@ -0,0 +1,11 @@
+[comment]: < Trust Lines Current -
+
+{% docs trust_lines_current %}
+
+The `trust_lines_current` table is a nightly snapshotted table that represents the current state of account trustlines. The table returns the latest trustline entry, per account, which is defined as the highest `last_modified_ledger` per trustline/account on the Stellar Network. Deleted trust lines are included in the table. For full state history, please use the `trust_lines` table.
+
+**The `trust_lines_current` table is only updated nightly. Intraday ledger state changes are not captured in the table.**
+
+As a reminder, trustline ledger entries store detailed information for assets trusted by a given account. The balance reported in the trustlines table reflects the account’s trusted asset balances, `XLM` balance is reported in the `accounts` table.
+
+{% enddocs %}

models/intermediate/trades/int_trade_agg_day.sql

@@ -1,5 +1,6 @@
 {{ config(
-    cluster_by =["asset_a", "asset_b"]
+    tags = ["trade_agg"]
+    , cluster_by =["asset_a", "asset_b"]
     )
 }}
 

models/intermediate/trades/int_trade_agg_month.sql

@@ -1,5 +1,6 @@
 {{ config(
-    cluster_by =["asset_a", "asset_b"]
+    tags = ["trade_agg"]
+    , cluster_by =["asset_a", "asset_b"]
     )
 }}
 

models/intermediate/trades/int_trade_agg_week.sql

@@ -1,5 +1,6 @@
 {{ config(
-    cluster_by =["asset_a", "asset_b"]
+    tags = ["trade_agg"]
+    , cluster_by =["asset_a", "asset_b"]
     )
 }}
 

models/intermediate/trades/int_trade_agg_year.sql

@@ -1,5 +1,6 @@
 {{ config(
-    cluster_by =["asset_a", "asset_b"]
+    tags = ["trade_agg"]
+    , cluster_by =["asset_a", "asset_b"]
     )
 }}
 

models/marts/enriched_history_operations.sql

@@ -1,5 +1,5 @@
 {{ config(
-    tags = ["partnership_assets", "asset_stats"],
+    tags = ["enriched_history_operations"],
     materialized='incremental',
     unique_key=["op_id"],
     partition_by={

models/marts/enriched_history_operations.yml

@@ -13,17 +13,6 @@ models:
           meta:
             description:
               "Monitors the freshness of your table over time, as the expected time between data updates."
-      - elementary.dimension_anomalies:
-          timestamp_column: closed_at
-          backfill_days: 90
-          dimensions:
-            - "type"
-          time_bucket:
-            period: day
-            count: 1
-          meta:
-            description:
-              "Monitors the frequency of values in the type column over time."
     columns:
       - name: op_id
         description: '{{ doc("operation_id") }}'

models/marts/fee_stats_agg.sql

@@ -1,4 +1,5 @@
 {{ config(
+    tags=["fee_stats"],
     materialized='incremental',
     unique_key=["day_agg"],
     partition_by={
@@ -164,4 +165,4 @@ with
     )
 
 select distinct *
-from renaming
+from renaming

models/marts/history_assets.sql

@@ -1,5 +1,6 @@
 {{ config(
-    materialized='incremental'
+    tags = ["history_assets"]
+    , materialized='incremental'
     , unique_key=["asset_id"]
     , cluster_by= ["asset_id"]
     )
@@ -81,4 +82,4 @@
 {% endif %}
 
 select *
-from add_assets
+from add_assets

models/marts/ledger_current_state/account_signers_current.sql

@@ -0,0 +1,62 @@
+{{ 
+    config(
+        tags = ["current_state"],
+        materialized = "incremental",
+        unique_key = "unique_id",
+        cluster_by = "account_id"
+    )
+}}
+
+/* Finds the latest state of each account signer in the `account_signers` table.
+   Ranks each record (grain: one row per account) using
+   last modified ledger sequence number. View includes all account signers.
+   (Deleted and Existing). View matches the Horizon snapshotted state tables. */
+with
+    current_signers as (
+        select
+            s.account_id
+            , s.signer
+            , s.weight
+            , s.sponsor
+            , s.last_modified_ledger
+            , l.closed_at
+            , s.ledger_entry_change
+            , s.deleted
+            -- table only has natural keys, creating a primary key
+            , concat(s.account_id, '-', s.signer
+            ) as unique_id
+            , s.batch_run_date
+            , s.batch_insert_ts
+            , row_number()
+                over (
+                    partition by s.account_id, s.signer
+                    order by s.last_modified_ledger desc, s.ledger_entry_change desc
+                ) as row_nr
+        from {{ ref('stg_account_signers') }} as s
+        join {{ ref('stg_history_ledgers') }} as l
+            on s.last_modified_ledger = l.sequence
+
+        {% if is_incremental() %}
+            -- limit the number of partitions fetched
+            where
+                s.batch_run_date >= date_sub(current_date(), interval 30 day)
+                -- fetch the last week of records loaded
+                and timestamp_add(s.batch_insert_ts, interval 7 day)
+                > (select max(t.upstream_insert_ts) from {{ this }} as t)
+        {% endif %}
+    )
+select
+    account_id
+    , signer
+    , weight
+    , sponsor
+    , last_modified_ledger
+    , ledger_entry_change
+    , closed_at
+    , deleted
+    , unique_id
+    , batch_run_date
+    , batch_insert_ts as upstream_insert_ts
+    , current_timestamp() as batch_insert_ts
+from current_signers
+where row_nr = 1

models/marts/ledger_current_state/account_signers_current.yml

@@ -0,0 +1,121 @@
+version: 2
+
+models:
+  - name: account_signers_current
+    description: '{{ doc("account_signers_current") }}'
+    tests:
+      - dbt_utils.recency:
+          datepart: hour
+          field: cast(closed_at as datetime)
+          interval: 12
+          config:
+            severity: warn
+          meta:
+            description:
+              "Monitors the freshness of your table over time, as the expected time between data updates."
+      - elementary.volume_anomalies:
+          timestamp_column: closed_at
+          backfill_days: 90
+          time_bucket:
+            period: day
+            count: 1
+          meta:
+            description:
+              "Monitors the row count of your table over time."
+      - dbt_utils.unique_combination_of_columns:
+          combination_of_columns:
+            - account_id
+            - signer
+            - ledger_entry_change
+            - last_modified_ledger
+          config:
+            where: closed_at >= timestamp_trunc(timestamp_sub(current_timestamp(), interval 2 day), day) 
+                    and closed_at < timestamp_trunc(current_timestamp(), day)
+          meta:
+            description:
+              "Tests the uniqueness combination of: account_id, signer, ledger_entry_change and last_modified_ledger."
+    columns:
+      - name: account_id
+        description: '{{ doc("account_id") }}'
+        tests:
+          - not_null:
+              config:
+                where: closed_at > current_timestamp - interval 2 day
+
+      - name: signer
+        description: '{{ doc("signer") }}'
+        tests:
+          - not_null:
+              config:
+                where: closed_at > current_timestamp - interval 2 day
+
+      - name: weight
+        description: '{{ doc("weight") }}'
+        tests:
+          - not_null:
+              config:
+                where: closed_at > current_timestamp - interval 2 day
+
+      - name: sponsor
+        description: '{{ doc("sponsor") }}'
+
+      - name: last_modified_ledger
+        description: '{{ doc("last_modified_ledger") }}'
+        tests:
+          - not_null:
+              config:
+                where: closed_at > current_timestamp - interval 2 day
+
+      - name: ledger_entry_change
+        description: '{{ doc("ledger_entry_change") }}'
+        tests:
+          - not_null:
+              config:
+                where: closed_at > current_timestamp - interval 2 day
+          - accepted_values:
+              values: [0, 1, 2]
+              quote: false
+
+      - name: closed_at
+        description: '{{ doc("closed_at") }}'
+        tests:
+          - not_null:
+              config:
+                where: closed_at > current_timestamp - interval 2 day
+
+      - name: deleted
+        description: '{{ doc("deleted") }}'
+        tests:
+          - not_null:
+              config:
+                where: closed_at > current_timestamp - interval 2 day
+          - accepted_values:
+              values: ["true", "false"]
+              quote: false
+
+      - name: batch_run_date
+        description: '{{ doc("batch_run_date") }}'
+        tests:
+          - not_null:
+              config:
+                where: closed_at > current_timestamp - interval 2 day
+
+      - name: batch_insert_ts
+        description: '{{ doc("batch_insert_ts") }}'
+        tests:
+          - not_null:
+              config:
+                where: closed_at > current_timestamp - interval 2 day
+
+      - name: upstream_insert_ts
+        description: '{{ doc("upstream_insert_ts") }}'
+        tests:
+          - not_null:
+              config:
+                where: closed_at > current_timestamp - interval 2 day  
+
+      - name: unique_id
+        description: '{{ doc("unique_id") }}'
+        tests:
+          - not_null
+          - unique