Bubblegum Update Metadata Version 1

[danenbm]

Overview

• This PR adds support for indexing the Bubblegum Update Metadata instruction on the mpl-bubblegum update-metadata-staging branch.

Detailed notes

• Similar to our previous approach for protecting against out-of-order/concurrent updates, it adds sequence numbers to the asset table and asset_data table for data that can be updated by both mint_v1 and update_metadata.
• To prevent overwriting data for decompressed assets, there were previously slot_updated checks on several pieces of data. These worked because the data was only written for compressed assets by mint_v1, and then after decompression only updated by Token Metadata indexing through account-based updates. In order to support update_metadata's ability to change some of the same asset data, these slot_updated checks were replaced with sequence number checks that allow for out-of-order indexing for compressed assets before and after they are decompressed.
• To handle multiple background tasks downloading off-chain metadata, a conditional check was added to the background task that makes sure the URI matches what is in the database before doing the http download (to avoid wasted work), and then again as a condition when updating the database with the downloaded data (to prevent updating the asset with data from a stale URI with concurrent updates). This fixes both a potential race condition with existing account-based updates for Token Metadata, and is also now required for compressed assets because update_metadata can change the URI and be indexed concurrently with mint_v1.
• Changes to the creator array for both compressed and non-compressed assets are now handled by upserting creators (using an ON CONFLICT with assetId and position columns). Bubblegum supports an empty creator array, and in that case a zero-length creator is inserted into the asset_creators table to allow for a sequence number (and thus handle out-of-order updates).
  • The reason the new approach does not delete existing creators and then re-insert updated creators, as was previously done with non-compressed assets, is that we found it could lead to race conditions where stale creators were inserted by concurrent updates.
  • To avoid the race condition we could have added an explicit lock or used the serializable transaction isolation level in Postgres. But to prevent indexing performance impacts we chose the no-lock solution with filtering on the API side.

Sea-ORM autogeneration

As mentioned offline, there were some slight discrepancies with the existing generated Sea-ORM code and the migrations, so I updated the generated code to match the migrations.

Alternate version

I created a Version 2 of this PR that overall uses less new sequence numbers, but required more refactoring and making sure mint_v1 and update_metadata both update all of the metadata fields.

The problem with Version 2 is that with multiple ingester processes, there can be race conditions. So sticking with this version, although incorporating some of the changes from Version 2

Regression testing

Tested with latest commit: 36df8e5

• Tested following instruction sequences from Bubblegum and verified database was in correct state afterwards:
  • create_tree, mint_v1, transfer, burn
  • create_tree, mint_v1, redeem, decompress_v1
  • create_tree, mint_v1, redeem, cancel_redeem, redeem, decompress_v1
  • create_tree, mint_v1, transfer, transfer
  • create_tree, mint_v1, delegate, transfer
  • create_tree, mint_v1, verify_creator
  • create_tree, mint_v1, verify_collection
  • create_tree, mint_v1, verify_collection, unverify_collection
  • create_tree, mint_v1, set_and_verify_collection
  • create_tree, mint_to_collection_v1, unverify_collection
• Deleted and rebuilt database, ran the instruction sequences from Bubblegum in reverse order (i.e.burn, transfer, mint_v1, create_tree, ) and verified database ended up in same state as when they were run forwards (decompressed items have slightly different expected state when run in reverse order).
• Also ran several of the sequences in non-sequential order and duplicated some of the transactions, and still got expected results.

Update Metadata testing

Tested with latest commit: 36df8e5

• Ran a createTree, mintV1, verifyCreator, unverifyCreator, and updateMetadata, and observed after the updateMetadata that the name, symbol, and URI had been changed.
• Deleted the database, then ran the exact same 5 transactions in reverse and observed exact same final result in the database.
• Again, ran a createTree, mintV1, verifyCreator, unverifyCreator, and updateMetadata, checking database state between each transaction.
  • Verified the second creator was set to verified after verifyCreator. Verified getAsset showed same result.
  • Verified the correct creator was set to unverified after unverifyCreator. Verified getAsset showed same result.
  • Verified that before updateMetadata, that getAssetsByCreator returned the asset when either of the two creators was used.
  • Verified that the name, symbol, URI, and creator array changed after updateMetadata, to the values specified in the updateArgs sent into the instruction.
  • Verified that after updateMetadata was sent, only the creator that was in the updateArgs was shown by a call to getAsset (the "stale" creator from asset_creators was NOT reported).
  • Also verified that after updateMetadata was sent, that getAssetsByCreator only returned the asset when the remaining creator was used. Using the "stale" creator from the database did not result in a returned asset.