Add mpl-core account indexing #178

danenbm · 2024-03-25T17:24:01Z

Notes

This indexes mpl-core Assets and Collections using account updates.
Two new interfaces were created: MplCoreAsset and MplCoreCollection to deal with fact that they are slightly different types and must be selectable by search_assets.
Added integration tests:
- getAsset for an mpl-core Asset.
- getAsset for an mpl-core Collection.
- getAssetsByAuthority
- getAssetsByGroup
- getAssetsByOwner

There's a couple scenarios which are hard to write integ tests for. These scenarios are described below and were tested locally with the Docker container:

Burned assets, which in the case of mpl-core, become 1-byte accounts but are never closed.
Assets with mpl-core plugins that are unknown to DAS.
- This is an anticipated case where the mpl-core contract is updated to have new plugins, but DAS has not been updated with the latest mpl-core Rust client.
- Rather than allowing DAS to fail, the output goes into an "unknown_plugins" section and is output as base64, which would allow users to parse it if they have an updated JS client. See below for an example.
- If the RPC provider wishes to later backfill assets that previously had uknown plugins, it can be done by selecting accounts where the unknown_plugins column is not NULL.

Related blockbuster PR

metaplex-foundation/blockbuster#41

Example output for an mpl-core `getAsset` with known and unknown plugins:

{
   "id" : 0,
   "jsonrpc" : "2.0",
   "result" : {
      "authorities" : [
         {
            "address" : "HBgemgDSTDJaKbkUJMabmUNrnPQcaovn9Vw4phJMiDMh",
            "scopes" : [
               "full"
            ]
         }
      ],
      "burnt" : false,
      "compression" : {
         "asset_hash" : "",
         "compressed" : false,
         "creator_hash" : "",
         "data_hash" : "",
         "eligible" : false,
         "leaf_id" : 0,
         "seq" : 0,
         "tree" : ""
      },
      "content" : {
         "$schema" : "https://schema.metaplex.com/nft1.0.json",
         "files" : [],
         "json_uri" : "https://example.com/asset",
         "links" : {},
         "metadata" : {
            "name" : "Test Asset",
            "symbol" : ""
         }
      },
      "creators" : [
         {
            "address" : "HBgemgDSTDJaKbkUJMabmUNrnPQcaovn9Vw4phJMiDMh",
            "share" : 100,
            "verified" : true
         }
      ],
      "grouping" : [
         {
            "group_key" : "collection",
            "group_value" : "5u2cpYZJYsL9gSCoLFAxTKpZTsCiyX6sJQKsRAr6epbL"
         }
      ],
      "id" : "315pzQSeaE6AQEtqVaFqR4mzSHxZSqXePVjQjWYKLWsW",
      "interface" : "MplCoreAsset",
      "mpl_core_info" : {
         "plugins_json_version" : 1
      },
      "mutable" : true,
      "ownership" : {
         "delegate" : "HBgemgDSTDJaKbkUJMabmUNrnPQcaovn9Vw4phJMiDMh",
         "delegated" : true,
         "frozen" : false,
         "owner" : "HBgemgDSTDJaKbkUJMabmUNrnPQcaovn9Vw4phJMiDMh",
         "ownership_model" : "single"
      },
      "plugins" : {
         "freeze_delegate" : {
            "authority" : {
               "address" : "HBgemgDSTDJaKbkUJMabmUNrnPQcaovn9Vw4phJMiDMh",
               "type" : "Address"
            },
            "data" : {
               "frozen" : false
            },
            "index" : 1,
            "offset" : 160
         },
         "royalties" : {
            "authority" : {
               "address" : null,
               "type" : "UpdateAuthority"
            },
            "data" : {
               "basis_points" : 5,
               "creators" : [
                  {
                     "address" : "HBgemgDSTDJaKbkUJMabmUNrnPQcaovn9Vw4phJMiDMh",
                     "percentage" : 100
                  }
               ],
               "rule_set" : "None"
            },
            "index" : 0,
            "offset" : 119
         },
         "transfer_delegate" : {
            "authority" : {
               "address" : null,
               "type" : "Owner"
            },
            "data" : {},
            "index" : 2,
            "offset" : 162
         }
      },
      "royalty" : {
         "basis_points" : 5,
         "locked" : false,
         "percent" : 0.0005,
         "primary_sale_happened" : false,
         "royalty_model" : "creators",
         "target" : null
      },
      "supply" : null,
      "unknown_plugins" : [
         {
            "authority" : {
               "address" : "HBgemgDSTDJaKbkUJMabmUNrnPQcaovn9Vw4phJMiDMh",
               "type" : "Address"
            },
            "data" : "CQA=",
            "index" : 3,
            "offset" : 163,
            "type" : 9
         }
      ]
   }
}

NicolasPennie · 2024-03-26T03:33:23Z

program_transformers/src/asset_upserts.rs

@@ -95,6 +96,11 @@ pub struct AssetMetadataAccountColumns {
    pub royalty_amount: i32,
    pub asset_data: Option<Vec<u8>>,
    pub slot_updated_metadata_account: u64,
+    pub plugins: Option<Value>,


Might be cleaner to store in a different table?

Yeah I could do that, its probably better then prefixing them with mpl_core_ in the main asset table.

Tough balance between normalization and query speed. Right now though the model is complex the read times are quick because its as simple index lookup. I'd like to keep that read speed.

The interface on the asset maps the plugins to a specific protocol if another pluggable asset type is creating in the future.

I am also happy keeping the new columns in the asset table. I do think it its better for the query speed.

I did draft it up using separate mpl_core table here: #180

As you guys have pointed out it will add the extra queries so we will skip it.

niks3089 · 2024-03-27T11:12:55Z

program_transformers/src/mpl_core_program/v1_asset.rs

+    let update_authority = match asset.update_authority {
+        UpdateAuthority::Address(address) => address.to_bytes().to_vec(),
+        UpdateAuthority::Collection(address) => {
+            find_collection_authority(conn, address.to_bytes().to_vec(), RETRY_INTERVALS)


I find the retry logic might not be a good idea here. We may slow down the indexing while retrying to index the account. Isn't this going to result in a race or may be a different state if collection authority isn't indexed yet?

yes it could slow down indexing, but honestly not sure how to deal with it other than a retry. This approach is similar to what token metadata indexing uses to look for the token.

In real life, for mpl-core the collection will have to exist before the assets are minted to the collection or if the asset is added to the collection, so this should not fail as long as concurrent process indexing got the collection.

niks3089 · 2024-03-27T11:17:12Z

program_transformers/src/mpl_core_program/v1_asset.rs

+        .one(conn)
+        .await?;
+
+    for interval in retry_intervals {


Should this loop be at the beginning? Seems like there's a sleep without actually fetching the authority after waking up. This will result in hitting the max retry every time

You are right I messed up this logic when copying it over from toke metadata and changing it. I think I was trying to make it more like a do-while but didn't complete it properly. Will fix now.

Fixed in #181

danenbm added 6 commits March 25, 2024 00:28

Add new db columns for mpl-core

4c669d5

Fix docker files

dfbf148

Regenerate SeaORM types

3841ea9

Add mpl-core ingestion

acb48c5

Add mpl-core DAS API output

086c47c

Add integ test for mpl-core asset and collection

4de7e5c

danenbm marked this pull request as ready for review March 26, 2024 01:28

danenbm added 3 commits March 25, 2024 19:00

add integ test for mpl-core getAssetsByGroup

88df0d7

add integ test for mpl-core getAssetsByAuthority

6642360

add integ test for mpl-core getAssetsByOwner

0c96dde

danenbm requested review from NicolasPennie, pmantica11, kespinola and fanatid March 26, 2024 02:59

danenbm mentioned this pull request Mar 26, 2024

Add mpl-core account parsing metaplex-foundation/blockbuster#41

Merged

NicolasPennie approved these changes Mar 26, 2024

View reviewed changes

danenbm mentioned this pull request Mar 26, 2024

feat: core indexing metaplex-foundation/aura#168

Merged

kespinola approved these changes Mar 27, 2024

View reviewed changes

niks3089 approved these changes Mar 27, 2024

View reviewed changes

danenbm added 2 commits March 27, 2024 02:29

Update to use published blockbuster crate

f412d12

Rename mpl-core collection current supply output to current size

2269810

danenbm force-pushed the danenbm/add-mpl-core branch from cdd6a48 to 2269810 Compare March 27, 2024 10:03

danenbm merged commit 2627055 into main Mar 27, 2024
3 checks passed

danenbm deleted the danenbm/add-mpl-core branch March 27, 2024 10:17

danenbm mentioned this pull request Mar 27, 2024

Use separate mpl_core table #180

Closed

niks3089 reviewed Mar 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add mpl-core account indexing #178

Add mpl-core account indexing #178

danenbm commented Mar 25, 2024 •

edited

Loading

NicolasPennie Mar 26, 2024

danenbm Mar 26, 2024

kespinola Mar 27, 2024

niks3089 Mar 27, 2024

danenbm Mar 27, 2024

niks3089 Mar 27, 2024

danenbm Mar 28, 2024

danenbm Mar 28, 2024

niks3089 Mar 27, 2024

danenbm Mar 28, 2024 •

edited

Loading

danenbm Mar 28, 2024

Add mpl-core account indexing #178

Add mpl-core account indexing #178

Conversation

danenbm commented Mar 25, 2024 • edited Loading

Notes

Related blockbuster PR

Example output for an mpl-core getAsset with known and unknown plugins:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danenbm Mar 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danenbm commented Mar 25, 2024 •

edited

Loading

Example output for an mpl-core `getAsset` with known and unknown plugins:

danenbm Mar 28, 2024 •

edited

Loading