Bounty 7: Problem to solve, Observation and Proposals

[piux2]

Problems to solve, Observation and Proposals

Part 1

TL;DR, I think we want to start with a pure current snapshot of the account state, and apply any changes post-facto.

part 1: given a block height (the latest blockheight that you know), export the current account state. I believe there is already some feature written to export state (and side note, it probably doesn't use RPC)--used to generate the genesis.json from cosmoshub-3. Test it out at a recent block height. Check it against production data; for example, does it show the amount of tokens held in IBC channels to osmosis? Upload the snapshot to S3 or some other file storage provider.

Observation: Verified.

Proposal:

gaiad export --height will export state in json. It can be used as a genesis state to bootstrap a node.
I found an exported genesis state snapshot for Vega upgrade testnet.

https://github.com/cosmos/vega-test/blob/66e7ccf559998d48a5bd230a3e6146bed856a83b/public-testnet/README.md#genesis-file

https://github.com/cosmos/vega-test/blob/66e7ccf559998d48a5bd230a3e6146bed856a83b/exported_unmodified_genesis.json.gz

    curl https://github.com/cosmos/vega-test/blob/66e7ccf559998d48a5bd230a3e6146bed856a83b/exported_unmodified_genesis.json.gz
    unzip ./exported_unmodified_genesis.json.gz

"genesis_time", 2019-12-11T16:11:34Z
"initial_height", 7,368,387

It was after the Delta (Gravity DEX) 13/07/21 6,910,000 cosmoshub-4 v0.34.x

Example to find balance on IBC

• find all accounts with ibc balances

    jq  '.app_state.bank.balances |select(.[].coins[].denom != "uatom")' exported_unmodified_genesis.json
  

• find accounts holding OSMO tokens

   jq  '.app_state.bank.balances[] |select(.coins[].denom == "ibc/14F9BC3E44B8A9C1BE1FB08980FAB87034C9905EF17CF2F5008FC085218811CC")' exported_unmodified_genesis.json
  

The token name and denom mapping can be found here token hub

To program fancy logic, we can use gojq
https://github.com/itchyny/gojq

Part 2

part 2: given an account A1 at a given block height in the past T1, and current block time T2, create a list of where all the account tokens are now, as a list of {Account;Coins} tuples. So if there were no transactions signed by A1 between T1 and T2, (and no unbondings before T1), the result would be simply [{A1;C1}] where C1 are the coins held by A1 at time T1. Implementation of this feature would start with SendTx, and then become staking aware. I don't know how best to do that off the top of my head. This also probably shouldn't use RPC but instead use go functions to iterate over blocks to avoid RPC overhead. That said, I might be wrong... if the RPC can handle say a month's worth of cosmoshub-4 blocks through localhost RPC in an hour, then it's fine. This might be feasible with unix pipes, or websockets.

Observation:
we can get current snapshot balances of all accounts from the exported state.

jq  '.app_state.bank.balances |select(.[].coins | length>0)' exported_unmodified_genesis.json

OR an account's balances.

   jq  '.app_state.bank.balances[] |select(.address=="cosmos1z4x4dyylwym26gnsjw29hqjhkwrw6vv2p6t58k")' exported_unmodified_genesis.json

The exported state does not contain user transactions during a period of time
Same for the delegation, it has the current delegation state of each account without the delegation records.

   jq  '.app_state.staking.delegations'  exported_unmodified_genesis.json > delegations.json

There are 171,027 accounts with balances

There are 144,197 records in the delegations

Because delegated tokens are not part of the account balance, we will need to merge these two together to calculate how many tokens each account owns.

Proposal:

We have two options

[ A ] write a program that just merges the two files in on Json that include delegation share as the additional coin in app_state.bank.balances.coins[]

Since the dataset is not huge, we merge it in memory. We can sort delegations array by delegations.delegator_address and balances array by address first. We loop through both arrays and put the result in a merged array.

The time complexity is O(N log N) for quicksort, as the address is quite random, and O(N) for merging
The space complexity is O(log N) for quicksort and O(1) for merging

PROS: simple

CONS: not flexible. we have to modify the code and get additional insights into the data set.

RESULTS: Less than 3 seconds

Joined and Merged 300,587 Accounts and 144,197 Delegations and tallied staking shares for less than 3s.

real 0m2.437s
user 0m2.085s
sys 0m0.600s

Will publish the source code soon

[ B ] dump it to two tables in postgreSQL and write the go program to query the database.

Once we dump data in postgreSQL as balances_table and delegation_table. we joined two tables, and add atoms amount and shares amount to get the total atoms that each account owns.

PROS: a lot more flexible to run SQL against the data once it is imported into the database, especially when we want to retrieve other insights from the same dataset.

CONS: complicated to set up at the beginning.

For this part of the requirement, it is the same as getting the current balance of each account.

So if there were no transactions signed by A1 between T1 and T2, (and no unbondings before T1), the result would be simply [{A1;C1}] where C1 are the coins held by A1 at time T1.

For this part of the requirement, it needs to calculate how many coins are added or removed from the account. Since the exported state file does not contains individual transactions. We can loop through send, delegation, and unbound messages from the state.db to a postgreSQL database and then query it.

given an account A1 at a given block height in the past T1, and current block time T2, create a list of where all the account tokens are now, as a list of {Account;Coins} tuples.

part 3

part 3: given a proposal, find all accounts that voted for/against/abstain, but also accounting for overrides by time (changing votes) and by delegation (overriding the validator's vote by a delegator).

Observations:
Votes from each account are not exported. Proposals only contain tallied results in the exported state file

jq  '.app_state.gov.votes' exported_unmodified_genesis.json
[]

Proposal:

We can loop through vote messages from state.db to a postgreSQL database and then query it.

[jaekwon]

Jacob, @LL-Cosmo, Can we identify the ICF's and AIB's addresses here? What are their amounts?

[jaekwon]

Also can somebody please write a short script to sum all the numbers, to verify against mintscan etc?

[LL-Cosmo]

Jacob, @LL-Cosmo, Can we identify the ICF's and AIB's addresses here? What are their amounts?

AIB Mintscan

• cosmos176m2p8l3fps3dal7h8gf9jvrv98tu3rqfdht86 Github
• Balance: ATOM 0.314218

ICF MULTISIG 1 Mintscan

• cosmos1unc788q8md2jymsns24eyhua58palg5kc7cstv Github
• Balance: ATOM 6,054,984.002730

ICF MULTISIG 2 Mintscan

• cosmos1z8mzakma7vnaajysmtkwt4wgjqr2m84tzvyfkz Github
• Balance: ATOM 7,809,184.455992

Source:
Cosmos Github
https://github.com/cosmos/mainnet/tree/master/accounts

[piux2]

Part 2 solution A is here

https://github.com/piux2/gnobounty7/blob/main/README2.md

The state data was exported at a height 7,368,387. 2021-08-20 08:58:06

Here is the merged results

AIB

     {
      "address": "cosmos176m2p8l3fps3dal7h8gf9jvrv98tu3rqfdht86",
      "coins": [
        {
          "amount": "310129",
          "denom": "uatom"
        },
        {
          "amount": "3687.255105864807774196",
          "denom": "shares"
        }
      ]
    }

Multisig 1

{
  "address": "cosmos1unc788q8md2jymsns24eyhua58palg5kc7cstv",
  "coins": [
    {
      "amount": "2210922814",
      "denom": "uatom"
    },
    {
      "amount": "6114291855534.014648437500000000",
      "denom": "shares"
    }
  ]
}

Multisig2

  {
    "address": "cosmos1z8mzakma7vnaajysmtkwt4wgjqr2m84tzvyfkz",
    "coins": [
      {
        "amount": "1257033924562",
        "denom": "uatom"
      },
      {
        "amount": "7165937142708.729492187500000000",
        "denom": "shares"
      }
    ]
  }

To verify it, we need a RPC endpoint that can return balance and delegations state at a height 7368387

   gaiad q bank balances cosmos1z8mzakma7vnaajysmtkwt4wgjqr2m84tzvyfkz --height 7368387

However, https://rpc.cosmos.network:443 does not have the state.