Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bubblegum Update Metadata Version 1 #134

Merged
merged 49 commits into from
Jan 9, 2024
Merged
Changes from 1 commit
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
09a599d
Add code to index Bubblegum Update Metadata
danenbm Oct 15, 2023
5ef2eb9
Update rust toolchain file
danenbm Oct 15, 2023
7dcf74b
Merge branch 'main' into danenbm/update-metadata-parsing
danenbm Oct 17, 2023
2084aa7
Fix moved variable after merge
danenbm Oct 17, 2023
269bf0d
Add code from mintV1 that allows for empty URI
danenbm Oct 17, 2023
bdf8e3c
Ordering using asset.seq initially applied to update_metadata
danenbm Oct 18, 2023
0240ce5
Add simple check for whether asset was decompressed to Bubblegum tran…
danenbm Oct 19, 2023
c26c693
Don't prevent sequence number update when already decompressed
danenbm Oct 19, 2023
dd891e8
Add sequence number to downloading metadata background task
danenbm Oct 19, 2023
97fe275
Add sequence number migration (Sea ORM not regenerated yet)
danenbm Oct 20, 2023
660a6ba
Regenerate Sea-ORM types
danenbm Oct 20, 2023
213dac5
Use new sequence numbers for Bubblegum Update Metadata
danenbm Oct 20, 2023
026c16a
Extra condition to protect out of order creator verification
danenbm Oct 21, 2023
e797adc
Remove base_info_seq for each creator and add creators_added_seq to a…
danenbm Oct 23, 2023
f2b1218
Regenerate Sea-ORM types
danenbm Oct 23, 2023
77020fa
Change creator metadata updates to use new creators_added_seq
danenbm Oct 23, 2023
ab1f1b4
Factor out common creator update code to helper function
danenbm Oct 23, 2023
3581269
Update to latest blockbuster beta
danenbm Dec 4, 2023
cb97203
Merge branch 'main' into danenbm/update-metadata-parsing
danenbm Dec 4, 2023
e62eb61
Use less than or equal for download metadata seq check
danenbm Dec 4, 2023
5808834
Index verified for token metadata collection
danenbm Dec 4, 2023
e38f64a
Add slot_updated to initial asset upsert, and removed duplicate items
danenbm Dec 4, 2023
933e299
Remove asset_was_decompressed
danenbm Dec 4, 2023
8732ba9
Rename royalty_amount_seq to base_info_seq
danenbm Dec 4, 2023
2500687
Fix typo in WHERE clause
danenbm Dec 4, 2023
fe1d522
Do not delete existing creators in mint_v1
danenbm Dec 4, 2023
d26fe00
Update comments around database txns
danenbm Dec 4, 2023
c4b1ab4
Use transaction in mint_V1 and update_metadata
danenbm Dec 4, 2023
9e305c9
Use transaction for other Bubblegum instructions asset table updates
danenbm Dec 4, 2023
c3798c0
Fix tree_id key index in update_metadata
danenbm Dec 4, 2023
7c0f9be
Remove use of was_decompressed flag on asset table
danenbm Dec 6, 2023
974966e
Add migration to remove was_decompressed and regenerate SeaORM types
danenbm Dec 6, 2023
6ccfb91
Combine upsert_asset_base_info and upsert_creators and add lock
danenbm Dec 9, 2023
b7c9ebe
Remove unneeded creators_added_seq
danenbm Dec 9, 2023
fa0839d
Switch to EXCLUSIVE mode lock
danenbm Dec 9, 2023
4b96e97
Add NULL condition check on asset.seq
danenbm Dec 13, 2023
8803152
Refactored creator indexing
danenbm Dec 18, 2023
1a80b98
Add conditions to creator upsert, add another check at DAS API level
danenbm Dec 18, 2023
2c14eeb
Rename asset_creators.verified_seq back to just regular seq
danenbm Dec 18, 2023
4c39726
Remove unneeded condition on asset_authority upsert
danenbm Dec 20, 2023
f669e76
Apply stale creator filtering to all DAS API queries
danenbm Dec 20, 2023
9155fa1
Use latest blockbuster beta release
danenbm Dec 21, 2023
6ccd27a
Remove download_metadata_seq and add URI match check instead
danenbm Dec 21, 2023
ef1f811
Fix task URI initial query
danenbm Dec 21, 2023
3dc3557
Regenerate Sea ORM types without download_metadata_seq
danenbm Dec 21, 2023
15d323e
Merge branch 'main' into danenbm/update-metadata-parsing
danenbm Dec 21, 2023
896af3d
asset_grouping.verified option remove
danenbm Dec 21, 2023
36df8e5
Fix filtering for getAssetsByCreator
danenbm Dec 21, 2023
c2875df
Update to blockbuster 0.9.0-beta.5 and mpl-bubblegum 1.0.1-beta.4
danenbm Jan 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Rename asset_creators.verified_seq back to just regular seq
danenbm committed Dec 18, 2023
commit 2c14eeb33f1355fa3ca5a4b166db037d96a20798
6 changes: 3 additions & 3 deletions digital_asset_types/src/dao/generated/asset_creators.rs
Original file line number Diff line number Diff line change
@@ -19,7 +19,7 @@ pub struct Model {
pub creator: Vec<u8>,
pub share: i32,
pub verified: bool,
pub verified_seq: Option<i64>,
pub seq: Option<i64>,
pub slot_updated: Option<i64>,
pub position: i16,
}
@@ -31,7 +31,7 @@ pub enum Column {
Creator,
Share,
Verified,
VerifiedSeq,
Seq,
SlotUpdated,
Position,
}
@@ -62,7 +62,7 @@ impl ColumnTrait for Column {
Self::Creator => ColumnType::Binary.def(),
Self::Share => ColumnType::Integer.def(),
Self::Verified => ColumnType::Boolean.def(),
Self::VerifiedSeq => ColumnType::BigInteger.def().null(),
Self::Seq => ColumnType::BigInteger.def().null(),
Self::SlotUpdated => ColumnType::BigInteger.def().null(),
Self::Position => ColumnType::SmallInteger.def(),
}
100 changes: 50 additions & 50 deletions digital_asset_types/src/dao/generated/sea_orm_active_enums.rs
Original file line number Diff line number Diff line change
@@ -7,35 +7,17 @@ use serde::{Deserialize, Serialize};
#[sea_orm(
rs_type = "String",
db_type = "Enum",
enum_name = "v1_account_attachments"
)]
pub enum V1AccountAttachments {
#[sea_orm(string_value = "edition")]
Edition,
#[sea_orm(string_value = "edition_marker")]
EditionMarker,
#[sea_orm(string_value = "master_edition_v1")]
MasterEditionV1,
#[sea_orm(string_value = "master_edition_v2")]
MasterEditionV2,
#[sea_orm(string_value = "unknown")]
Unknown,
}
#[derive(Debug, Clone, PartialEq, EnumIter, DeriveActiveEnum, Serialize, Deserialize)]
#[sea_orm(
rs_type = "String",
db_type = "Enum",
enum_name = "specification_versions"
enum_name = "royalty_target_type"
)]
pub enum SpecificationVersions {
pub enum RoyaltyTargetType {
#[sea_orm(string_value = "creators")]
Creators,
#[sea_orm(string_value = "fanout")]
Fanout,
#[sea_orm(string_value = "single")]
Single,
#[sea_orm(string_value = "unknown")]
Unknown,
#[sea_orm(string_value = "v0")]
V0,
#[sea_orm(string_value = "v1")]
V1,
#[sea_orm(string_value = "v2")]
V2,
}
#[derive(Debug, Clone, PartialEq, EnumIter, DeriveActiveEnum, Serialize, Deserialize)]
#[sea_orm(rs_type = "String", db_type = "Enum", enum_name = "mutability")]
@@ -76,42 +58,60 @@ pub enum SpecificationAssetClass {
Unknown,
}
#[derive(Debug, Clone, PartialEq, EnumIter, DeriveActiveEnum, Serialize, Deserialize)]
#[sea_orm(rs_type = "String", db_type = "Enum", enum_name = "task_status")]
pub enum TaskStatus {
#[sea_orm(string_value = "failed")]
Failed,
#[sea_orm(string_value = "pending")]
Pending,
#[sea_orm(string_value = "running")]
Running,
#[sea_orm(string_value = "success")]
Success,
#[sea_orm(rs_type = "String", db_type = "Enum", enum_name = "owner_type")]
pub enum OwnerType {
#[sea_orm(string_value = "single")]
Single,
#[sea_orm(string_value = "token")]
Token,
#[sea_orm(string_value = "unknown")]
Unknown,
}
#[derive(Debug, Clone, PartialEq, EnumIter, DeriveActiveEnum, Serialize, Deserialize)]
#[sea_orm(
rs_type = "String",
db_type = "Enum",
enum_name = "royalty_target_type"
enum_name = "v1_account_attachments"
)]
pub enum RoyaltyTargetType {
#[sea_orm(string_value = "creators")]
Creators,
#[sea_orm(string_value = "fanout")]
Fanout,
#[sea_orm(string_value = "single")]
Single,
pub enum V1AccountAttachments {
#[sea_orm(string_value = "edition")]
Edition,
#[sea_orm(string_value = "edition_marker")]
EditionMarker,
#[sea_orm(string_value = "master_edition_v1")]
MasterEditionV1,
#[sea_orm(string_value = "master_edition_v2")]
MasterEditionV2,
#[sea_orm(string_value = "unknown")]
Unknown,
}
#[derive(Debug, Clone, PartialEq, EnumIter, DeriveActiveEnum, Serialize, Deserialize)]
#[sea_orm(rs_type = "String", db_type = "Enum", enum_name = "owner_type")]
pub enum OwnerType {
#[sea_orm(string_value = "single")]
Single,
#[sea_orm(string_value = "token")]
Token,
#[sea_orm(
rs_type = "String",
db_type = "Enum",
enum_name = "specification_versions"
)]
pub enum SpecificationVersions {
#[sea_orm(string_value = "unknown")]
Unknown,
#[sea_orm(string_value = "v0")]
V0,
#[sea_orm(string_value = "v1")]
V1,
#[sea_orm(string_value = "v2")]
V2,
}
#[derive(Debug, Clone, PartialEq, EnumIter, DeriveActiveEnum, Serialize, Deserialize)]
#[sea_orm(rs_type = "String", db_type = "Enum", enum_name = "task_status")]
pub enum TaskStatus {
#[sea_orm(string_value = "failed")]
Failed,
#[sea_orm(string_value = "pending")]
Pending,
#[sea_orm(string_value = "running")]
Running,
#[sea_orm(string_value = "success")]
Success,
}
#[derive(Debug, Clone, PartialEq, EnumIter, DeriveActiveEnum, Serialize, Deserialize)]
#[sea_orm(rs_type = "String", db_type = "Enum", enum_name = "chain_mutability")]
6 changes: 3 additions & 3 deletions digital_asset_types/src/dao/scopes/asset.rs
Original file line number Diff line number Diff line change
@@ -329,16 +329,16 @@ pub async fn get_by_id(
// Any creators that are not the max seq are stale rows or updated by Token Metadata (seq = 0), so remove them.
let seq = if creators
.iter()
.map(|creator| creator.verified_seq)
.map(|creator| creator.seq)
.any(|seq| seq == Some(0))
{
Some(Some(0))
} else {
creators.iter().map(|creator| creator.verified_seq).max()
creators.iter().map(|creator| creator.seq).max()
};

if let Some(seq) = seq {
creators.retain(|creator| creator.verified_seq == seq);
creators.retain(|creator| creator.seq == seq);
}
}

2 changes: 1 addition & 1 deletion digital_asset_types/tests/common.rs
Original file line number Diff line number Diff line change
@@ -184,7 +184,7 @@ pub fn create_asset_creator(
creator,
share,
verified,
verified_seq: Some(0),
seq: Some(0),
slot_updated: Some(0),
position: 0,
},
Original file line number Diff line number Diff line change
@@ -10,18 +10,6 @@ pub struct Migration;
#[async_trait::async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
manager
.get_connection()
.execute(Statement::from_string(
DatabaseBackend::Postgres,
"
ALTER TABLE asset_creators
RENAME COLUMN seq to verified_seq;
"
.to_string(),
))
.await?;

manager
.alter_table(
Table::alter()
@@ -45,18 +33,6 @@ impl MigrationTrait for Migration {
}

async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
manager
.get_connection()
.execute(Statement::from_string(
DatabaseBackend::Postgres,
"
ALTER TABLE asset_creators
RENAME COLUMN verified_seq to seq;
"
.to_string(),
))
.await?;

manager
.alter_table(
Table::alter()
12 changes: 6 additions & 6 deletions nft_ingester/src/program_transformers/bubblegum/db.rs
Original file line number Diff line number Diff line change
@@ -567,12 +567,12 @@ where
// for the creator.
db_creators.push(asset_creators::ActiveModel {
asset_id: Set(id.clone()),
creator: Set(vec![]),
position: Set(0),
creator: Set(vec![]),
share: Set(100),
slot_updated: Set(Some(slot_updated)),
verified: Set(false),
verified_seq: Set(Some(seq)),
slot_updated: Set(Some(slot_updated)),
seq: Set(Some(seq)),
..Default::default()
});
} else {
@@ -590,8 +590,8 @@ where
creator: Set(c.address.to_bytes().to_vec()),
share: Set(c.share as i32),
verified: Set(c.verified),
verified_seq: Set(Some(seq)),
slot_updated: Set(Some(slot_updated)),
seq: Set(Some(seq)),
..Default::default()
});

@@ -610,15 +610,15 @@ where
asset_creators::Column::Creator,
asset_creators::Column::Share,
asset_creators::Column::Verified,
asset_creators::Column::VerifiedSeq,
asset_creators::Column::SlotUpdated,
asset_creators::Column::Seq,
])
.to_owned(),
)
.build(DbBackend::Postgres);

query.sql = format!(
"{} WHERE (asset_creators.verified_seq != 0 AND excluded.verified_seq >= asset_creators.verified_seq) OR asset_creators.verified_seq IS NULL",
"{} WHERE (asset_creators.seq != 0 AND excluded.seq >= asset_creators.seq) OR asset_creators.seq IS NULL",
query.sql
);

Original file line number Diff line number Diff line change
@@ -343,8 +343,8 @@ pub async fn save_v1_asset<T: ConnectionTrait + TransactionTrait>(
creator: Set(c.address.to_bytes().to_vec()),
share: Set(c.share as i32),
verified: Set(c.verified),
verified_seq: Set(Some(0)),
slot_updated: Set(Some(slot_i)),
seq: Set(Some(0)),
..Default::default()
});
creators_set.insert(c.address);
@@ -374,8 +374,8 @@ pub async fn save_v1_asset<T: ConnectionTrait + TransactionTrait>(
asset_creators::Column::Creator,
asset_creators::Column::Share,
asset_creators::Column::Verified,
asset_creators::Column::VerifiedSeq,
asset_creators::Column::SlotUpdated,
asset_creators::Column::Seq,
])
.to_owned(),
)