P21 Field Addition to the Staging Layer (#71)

* - Updated documentation for the fields added
- Linting changes
- Modified the staging Models to account for the new fields added after P 21 Upgrade

* - Updated documentation for the fields added
- Linting changes
- Modified the staging Models to account for the new fields added after P 21 Upgrade

* - Empty Commit after resolving Merge Conflicts

* Updated the indentation changes to adhere to linting standards

* Addressed the review comments on the field descriptions

models/docs/sources/contract_code.md

@@ -9,3 +9,43 @@ _Note:_ contract code does not contain the actual wasm. The physical wasm can be
 {% docs contract_code_ext_v %}
 Contract code extention version
 {% enddocs %}
+
+{% docs n_data_segment_bytes %}
+The total number of bytes used by the data segments in the contract code. Data segments are used to initialize the linear memory of a WebAssembly (WASM) module.
+{% enddocs %}
+
+{% docs n_data_segments %}
+The number of data segments in the contract code. Data segments define the sections of memory that are initialized with specific values when the WebAssembly module is instantiated.
+{% enddocs %}
+
+{% docs n_elem_segments %}
+The number of element segments in the contract code. Element segments are used to initialize the tables in a WebAssembly (WASM) module, typically used for functions.
+{% enddocs %}
+
+{% docs n_exports %}
+The number of exports in the contract code. Exports are functions, memories, tables, or globals that are made available to the host environment by the WebAssembly module.
+{% enddocs %}
+
+{% docs n_functions %}
+The number of functions defined in the contract code. This includes all the functions that can be executed within the WebAssembly (WASM) module.
+{% enddocs %}
+
+{% docs n_globals %}
+The number of global variables in the contract code. Globals are variables that hold a single value and can be accessed and modified by the WebAssembly module.
+{% enddocs %}
+
+{% docs n_imports %}
+The number of imports in the contract code. Imports are functions, memories, tables, or globals that the WebAssembly module requires from the host environment.
+{% enddocs %}
+
+{% docs n_instructions %}
+The total number of instructions in the contract code. Instructions are the individual operations that the WebAssembly module can execute.
+{% enddocs %}
+
+{% docs n_table_entries %}
+The number of entries in the tables defined by the contract code. Tables are used to store function references in WebAssembly (WASM) modules.
+{% enddocs %}
+
+{% docs n_types %}
+The number of types defined in the contract code. Types define the signatures of functions, including the number and type of arguments and return values.
+{% enddocs %}

models/docs/sources/contract_data.md

@@ -22,3 +22,23 @@ Contract can either be temporary or persistent
 {% docs balance_holder %}
 The address/account that holds the balance of the asset in contract data
 {% enddocs %}
+
+{% docs key %}
+The encoded key used to identify a specific piece of contract data. The encoded key has two components (type and value) where type describes the data type and the value describes the encoded value of the data type for the contract data.
+{% enddocs %}
+
+{% docs key_decoded %}
+The human-readable or decoded version of the key. This provides an understandable format of the key, making it easier to interpret and use in analysis.
+{% enddocs %}
+
+{% docs val %}
+The encoded value associated with the key in the contract data. The encoded val has two components (type and value) where type describes the data type and the value describes the encoded value of the data type for the contract data.
+{% enddocs %}
+
+{% docs val_decoded %}
+The human-readable or decoded version of the value. This provides a clear and understandable representation of the value, making it easier to interpret and use in analysis.
+{% enddocs %}
+
+{% docs contract_data_xdr %}
+The XDR (External Data Representation) encoding of the contract data. XDR is a standard format used to serialize and deserialize data, ensuring interoperability across different systems. This field contains the raw, serialized contract data in XDR format.
+{% enddocs %}

models/docs/sources/history_effects.md

@@ -143,3 +143,11 @@ Soroban contract invoke host function event type: transfer, mint, clawback, or b
 {% docs details_entries %}
 Soroban ledger change entries related to contract extension
 {% enddocs %}
+
+{% docs id %}
+This unique effect id is the concatenation of the operation id and the effect index
+{% enddocs %}
+
+{% docs index %}
+The index of the effect within the transaction. This index helps to order effects that result from the same transaction, indicating the sequence in which they occurred.
+{% enddocs %}

models/docs/sources/history_ledgers.md

@@ -110,13 +110,13 @@ The total number of operations in the transaction set for this ledger, including
 {% enddocs %}
 
 {% docs soroban_fee_write_1kb %}
-The write fee is dynamic fee that grows linearly with the size of the ledger. The write fee for a given ledger is recorded in the ledger header and has a direct relationship to the bucketlist size of the ledger db.
+The fee associated with writing 1KB of data to the Soroban ledger.The write fee is dynamic fee that grows linearly with the size of the ledger. The write fee for a given ledger is recorded in the ledger header and has a direct relationship to the bucketlist size of the ledger db.
 {% enddocs %}
 
 {% docs node_id %}
 The id of winning validator node which is allowed to write transaction set to the network. The winning validator is decided by the network.
 {% enddocs %}
 
 {% docs signature %}
-The signing hash of the validator node which writes the transaction set to the network.
+The signing hash of the validator node which writes the transaction set to the network.This signature ensures the integrity and authenticity of the ledger, confirming that it has not been tampered with.
 {% enddocs %}

models/docs/sources/history_operations.md

@@ -1404,17 +1404,6 @@ _Note:_ Only applies to Stellar Asset Contracts
 
 {% enddocs %}
 
-{% docs details_extend_to %}
-Soroban field - ledger extended to
-
-### Only exists for the following operations:
-
-| Type | Operation            |
-| ---- | -------------------- |
-| 25   | Extend Footprint Ttl |
-
-{% enddocs %}
-
 {% docs details_asset_balance_changes %}
 Soroban field - asset balance changes from invoke host function
 
@@ -1466,3 +1455,27 @@ Soroban field - soroban invoke function (invoke contract, create contract, uploa
 | 24   | Invoke Host Function |
 
 {% enddocs %}
+
+{% docs details_asset_id %}
+The identifier of the asset involved in the operation. This field provides details about the specific asset being operated on, such as in payment or transfer operations.
+{% enddocs %}
+
+{% docs details_buying_asset_id %}
+The identifier of the asset being bought in the operation. This is relevant for trade and offer operations, where one asset is exchanged for another.
+{% enddocs %}
+
+{% docs details_selling_asset_id %}
+The identifier of the asset being sold in the operation. This field is used in trade and offer operations, where one asset is exchanged for another.
+{% enddocs %}
+
+{% docs details_source_asset_id %}
+The identifier of the source asset in the operation. This is used in payment operations where an asset is transferred or converted from one form to another.
+{% enddocs %}
+
+{% docs details_reserve_a_asset_id %}
+The identifier of the first reserve asset in a liquidity pool or similar construct. This field is relevant for operations involving liquidity pools, where multiple assets are pooled together.
+{% enddocs %}
+
+{% docs details_reserve_b_asset_id %}
+The identifier of the second reserve asset in a liquidity pool or similar construct. This field complements `details.reserve_a_asset_id` in describing the assets involved in a liquidity pool.
+{% enddocs %}

models/docs/sources/history_transactions.md

@@ -17,13 +17,6 @@ A unique identifier for this transaction.
 {% docs transaction_hash %}
 A hex-encoded SHA-256 hash of this transaction's XDR-encoded form.
 
-- Required Field
-  {% enddocs %}
-
-{% docs ledger_sequence %}
-The sequence number of the ledger that this transaction was included in.
-
-- Cluster Field
 - Required Field
   {% enddocs %}
 
@@ -211,3 +204,23 @@ The fee charged for the transaction to be included in the ledger.
 {% docs resource_fee_refund %}
 The amount of the resource fee refunded to the transaction submitter. The refundable fees are calculated from rent, events and return value. Refundable fees are charged from teh source account before the transaction is executed and then refunded based on the actual usage.
 {% enddocs %}
+
+{% docs tx_signers %}
+The public keys of the signers who authorized the transaction. This field lists all the signatories that validated and approved the transaction, ensuring it meets the required authorization thresholds.
+{% enddocs %}
+
+{% docs non_refundable_resource_fee_charged %}
+The amount of the resource fee charged for the transaction that is non-refundable. This fee is deducted from the transaction initiator's account and is not returned, regardless of the transaction's success or failure. It covers the cost of network resources consumed.
+{% enddocs %}
+
+{% docs refundable_resource_fee_charged %}
+The amount of the resource fee charged for the transaction that is refundable. This fee may be returned to the transaction initiator if certain conditions are met, such as the transaction failing to be included in a ledger.
+{% enddocs %}
+
+{% docs rent_fee_charged %}
+The fee charged for renting resources on the network, such as storage space for data. This fee is deducted from the transaction initiator's account and is non-refundable, covering the ongoing cost of maintaining the data on the network. A rent fee wouldn't be charged on failed transactions.
+{% enddocs %}
+
+{% docs refundable_fee %}
+The portion of the transaction fee that is refundable under certain conditions. This field indicates the amount that can be returned to the transaction initiator if the transaction does not fully execute or meets specific refund criteria.
+{% enddocs %}

models/docs/universal.md

@@ -99,6 +99,18 @@ The timestamp in UTC when the ledger with this trade was closed.
 Soroban contract id
 {% enddocs %}
 
+
+{% docs details_extend_to %}
+Soroban field - ledger extended to
+
+### Only exists for the following operations:
+
+| Type | Operation            |
+| ---- | -------------------- |
+| 25   | Extend Footprint Ttl |
+
+{% enddocs %}
+
 {% docs contract_code_hash %}
 Soroban contract code hash
 {% enddocs %}
@@ -110,3 +122,10 @@ Hash of the ledgerKey which is a subset of the ledgerEntry. The subset of ledger
 {% docs closed_at %}
 Timestamp in UTC when this ledger closed and committed to the network. Ledgers are expected to close ~every 5 seconds
 {% enddocs %}
+
+{% docs ledger_sequence %}
+The sequence number of this ledger. It represents the order of the ledger within the Stellar blockchain. Each ledger has a unique sequence number that increments with every new ledger, ensuring that ledgers are processed in the correct order.
+
+- Cluster Field
+- Required Field
+{% enddocs %}

models/marts/history_assets.sql

@@ -31,8 +31,9 @@
                 , new_load.asset_type
             from new_load
             left join
-                deduplicated_table on
-            new_load.asset_id = deduplicated_table.asset_id
+                deduplicated_table
+                on
+                new_load.asset_id = deduplicated_table.asset_id
             where
                 deduplicated_table.asset_id is null
         )

models/marts/ledger_current_state/config_settings_current.sql

@@ -64,10 +64,10 @@ with
             , cfg.batch_insert_ts
             , cfg.ledger_sequence
             , row_number()
-            over (
-                partition by cfg.config_setting_id
-                order by cfg.closed_at desc
-            ) as rn
+                over (
+                    partition by cfg.config_setting_id
+                    order by cfg.closed_at desc
+                ) as rn
         from {{ ref('stg_config_settings') }} as cfg
         {% if is_incremental() %}
             -- limit the number of partitions fetched incrementally

models/marts/ledger_current_state/contract_code_current.sql

@@ -25,10 +25,10 @@ with
             , cc.ledger_sequence
             , cc.ledger_key_hash
             , row_number()
-            over (
-                partition by cc.contract_code_hash
-                order by cc.closed_at desc
-            ) as rn
+                over (
+                    partition by cc.contract_code_hash
+                    order by cc.closed_at desc
+                ) as rn
         from {{ ref('stg_contract_code') }} as cc
         {% if is_incremental() %}
             -- limit the number of partitions fetched incrementally

models/marts/ledger_current_state/contract_data_current.sql

@@ -32,10 +32,10 @@ with
             , cd.ledger_key_hash
             , concat(cd.contract_id, '-', cd.ledger_key_hash) as unique_id
             , row_number()
-            over (
-                partition by cd.contract_id, cd.ledger_key_hash
-                order by cd.closed_at desc
-            ) as rn
+                over (
+                    partition by cd.contract_id, cd.ledger_key_hash
+                    order by cd.closed_at desc
+                ) as rn
         from {{ ref('stg_contract_data') }} as cd
         {% if is_incremental() %}
             -- limit the number of partitions fetched incrementally

models/marts/ledger_current_state/ttl_current.sql

@@ -23,10 +23,10 @@ with
             , ttl.batch_run_date
             , ttl.batch_insert_ts
             , row_number()
-            over (
-                partition by ttl.key_hash
-                order by ttl.closed_at
-            ) as rn
+                over (
+                    partition by ttl.key_hash
+                    order by ttl.closed_at
+                ) as rn
         from {{ ref('stg_ttl') }} as ttl
         {% if is_incremental() %}
             -- limit the number of partitions fetched incrementally

models/staging/stg_claimable_balances.sql

@@ -16,6 +16,7 @@ with
             , asset_type
             , asset_code
             , asset_issuer
+            , asset_id
             , asset_amount
             , sponsor
             , flags

models/staging/stg_contract_code.sql

@@ -17,6 +17,16 @@ with
             , ledger_entry_change
             , ledger_sequence
             , ledger_key_hash
+            , n_data_segment_bytes
+            , n_data_segments
+            , n_elem_segments
+            , n_exports
+            , n_functions
+            , n_globals
+            , n_imports
+            , n_instructions
+            , n_table_entries
+            , n_types
             , closed_at
             , deleted
             , batch_id

models/staging/stg_contract_data.sql

@@ -23,6 +23,11 @@ with
             , ledger_entry_change
             , ledger_sequence
             , ledger_key_hash
+            , key
+            , key_decoded
+            , val
+            , val_decoded
+            , contract_data_xdr
             , closed_at
             , deleted
             , batch_id

models/staging/stg_history_effects.sql

@@ -11,7 +11,9 @@ with
 
     , history_effects as (
         select
-            address
+            id
+            , ledger_sequence
+            , address
             , address_muxed
             , operation_id
             , type
@@ -75,6 +77,8 @@ with
             , details.contract
             , details.ledgers_to_expire
             , details.entries
+            , details.extend_to
+            , index
             , closed_at
             , batch_id
             , batch_run_date

models/staging/stg_history_ledgers.sql

@@ -18,6 +18,7 @@ with
             , operation_count as ledger_operation_count
             , closed_at
             , id as ledger_id
+            , node_id
             , total_coins
             , fee_pool
             , base_fee
@@ -28,6 +29,8 @@ with
             , successful_transaction_count
             , failed_transaction_count
             , tx_set_operation_count
+            , signature
+            , soroban_fee_write_1kb
             , batch_id
             , batch_run_date
             , batch_insert_ts