WIP : optimize search assets query #230
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
Nagaprasadvr
commented
Dec 10, 2024
Nagaprasadvr
commented
- Add composite indexes on asset table
- Add composite indexes on asset_grouping table
- drop order by for non-numeric fields which increased latency
… single. Logic in token account updates would change owner when any token account had amount > 0 would triggers spam updates of the owner of the asset with any transfer. (#112)
…_tree_get_size errors from not being able to determine the trees size. this immediately ends the loop and logs the error preventing trees from being gathered. instead continue to the next tree and exclude it from the trees to backfill. (#113)
* refactor(backfiller): tree backfilling using getSignaturesForAdress. fetch all trees, fetch associated transactions * feat(backfiller): generate table and model for query last transaction record for fast forwarding tree transaction crawling * feat(backfiller): push transaction payloads to redis through the perkle messenger. mark tree transactons as processed_at so know it completed the index loop. * fix(backfiller): git history changes made from running formatter. just include changes needed by the backfiller. * fix(backfiller): support mock feature for sea-orm by switching to pg pool and sea_orm adapter.
* refactor(backfiller): tree backfilling using getSignaturesForAdress. fetch all trees, fetch associated transactions * feat(backfiller): generate table and model for query last transaction record for fast forwarding tree transaction crawling * feat(backfiller): push transaction payloads to redis through the perkle messenger. mark tree transactons as processed_at so know it completed the index loop. * fix(backfiller): git history changes made from running formatter. just include changes needed by the backfiller. * fix(backfiller): support mock feature for sea-orm by switching to pg pool and sea_orm adapter. * feat(backfiller): emit performance and reliability metrics to statsd * refactor(backfiller): only backfill trees from finalized blocks * refactor(backfiller): skip queueing a transaction that is already processed at
…ig saved in the db. also drop senders so program can close. (#116)
…rom a queue channel to a pool. move db checks for inserting a transaction into transaction worker thread instead of tree thread allowing for concurrent saving of transactions. (#117)
…info tracked by cl_items and cl_audits (#119)
…que constraint is in place
* feat: find gaps in trees * refactor(backfiller): search for gaps in cl_audits_v2. add worker queue for gaps so crawling done concurrently.
…agged for reindex. in worker process fetch the metadata json and update the record setting reindex to false. emit metrics and logg error or success. (#122)
* feat(metadata_json): port the backfill command. refactor to a worker struct to be used by the backfill and ingest commands. setup receiver for sending assets that need to be indexed to a redis stream. * feat(metadata_json): add new crate for continuesly process messages and do backfill.
…n. general refactor to improve readability of the crate. (#124)
* Update raw name and raw symbol for existing NFTs (#139) Update the raw name and raw symbol from the onchain data for existing NFTs when reingesting. This allows correction of incorrect values during reprocessing on an existing index. * Upstream Helius features (#133) * Bubblegum Update Metadata Version 1 (#134) * Add code to index Bubblegum Update Metadata * Update rust toolchain file * Fix moved variable after merge * Add code from mintV1 that allows for empty URI * Ordering using asset.seq initially applied to update_metadata * Add simple check for whether asset was decompressed to Bubblegum transformers * Don't prevent sequence number update when already decompressed * Add sequence number to downloading metadata background task * Add sequence number migration (Sea ORM not regenerated yet) * Regenerate Sea-ORM types * Use new sequence numbers for Bubblegum Update Metadata * Extra condition to protect out of order creator verification * Remove base_info_seq for each creator and add creators_added_seq to asset table * Regenerate Sea-ORM types * Change creator metadata updates to use new creators_added_seq * Factor out common creator update code to helper function * Update to latest blockbuster beta * Use less than or equal for download metadata seq check * Index verified for token metadata collection * Add slot_updated to initial asset upsert, and removed duplicate items * Remove asset_was_decompressed Replaced with WHERE clauses on each upsert. Move remaining upserts from mint_v1 to db.rs. Remove upsert to asset_v1_account_attachments from mint_V1. Combine upserts for asset base info and royalty amount. * Rename royalty_amount_seq to base_info_seq * Fix typo in WHERE clause * Do not delete existing creators in mint_v1 * Update comments around database txns * Use transaction in mint_V1 and update_metadata * Use transaction for other Bubblegum instructions asset table updates * Fix tree_id key index in update_metadata * Remove use of was_decompressed flag on asset table * Add migration to remove was_decompressed and regenerate SeaORM types * Combine upsert_asset_base_info and upsert_creators and add lock * Remove unneeded creators_added_seq * Switch to EXCLUSIVE mode lock * Add NULL condition check on asset.seq * Refactored creator indexing * Use new Blockbuster that always updates all creators and verification status. * Remove deleting creators with lower sequence numbers as it would not work due to race conditions. * Add concept of "empty" creator value to support Bubblegum empty creator arrays. * Add filtering out of old creators or having no creators to DAS code. * Also get authority and tree_id accounts from Bubblegum during mint and update_metadata. * Add conditions to creator upsert, add another check at DAS API level * Rename asset_creators.verified_seq back to just regular seq * Remove unneeded condition on asset_authority upsert * Apply stale creator filtering to all DAS API queries * Use latest blockbuster beta release * Remove download_metadata_seq and add URI match check instead * Fix task URI initial query * Regenerate Sea ORM types without download_metadata_seq * asset_grouping.verified option remove * Fix filtering for getAssetsByCreator * Update to blockbuster 0.9.0-beta.5 and mpl-bubblegum 1.0.1-beta.4 * Configurable account streams (#148) * Make workers configurable Make workers fully configurab le and remove reference to the plerkle plugin. * fix lifetime --------- Co-authored-by: Kirill Fomichev <[email protected]> * fix: splt tokens with no token stanard are incorrectly categorized as single. Logic in token account updates would change owner when any token account had amount > 0 would triggers spam updates of the owner of the asset with any transfer. (#151) * Improve workspace usage (#141) * fix: remove decompress ix handling and use db transactions (#156) * Improvements for consistency NFT indexing and query consistency (#157) * Danenbm/bubblegum sequence tests 2 (#160) * Add script to forward transactions an check database results * Fix ordering and add debug info * Add remaining non-creator/non-collection tests * Require asset and cl_items files to exist * Add asset_creators and asset_grouping tests * Add verify_creator and verify_collection tests * Add more collection verification tests * Move test data to subirectory * Move repeated code to functions * Add support for running sequences in reverse * Add instructions to README for running test script * Minor README update * feat: `getSignaturesForAsset` endpoint on top of new `cl_audits_v2` table (#155) * add migration files for cl_audits_v2 * add types * ingester * add getSignaturesForAsset endpoint * refactor to resolve merge conflict related bugs * address clippy error * rename to get_asset_signatures * add instruction type update_metadata * add error log if instruction is unknown * add sort order changes --------- Co-authored-by: Nicolas Pennie <[email protected]> * fix ORM circular dependency (#161) * fix ORM circular dependency * PR comments --------- Co-authored-by: Linus Kendall <[email protected]> Co-authored-by: Tahsin Tunan <[email protected]> Co-authored-by: Michael Danenberg <[email protected]> Co-authored-by: Kirill Fomichev <[email protected]> Co-authored-by: Nicolas Pennie <[email protected]>
…e signature and transaction log for every change
…ree backfilling and causing excessive locks on db slowing ingestion processing (#128)
…he main thread for a mint (#129)
…_v2. requires rerun of the backfiller. if log-path passed pick back up. switch to saving cl_audits_v2 id as the entry of the audit to allow for picking back work. (#132)
* patch remove slot field check from asset update * Mimic null check on the asset upsert * correct where clause for creators on conflicts * add owner_type signle check to the owner update on asset * Add field noop checks on mpl and asset upserts
…ub.com:rpcpool/digital-asset-rpc-infrastructure into eclipse-build
…asset-id' of github.com:rpcpool/digital-asset-rpc-infrastructure into eclipse-build
…ets fun bug fixed
…of github.com:rpcpool/digital-asset-rpc-infrastructure into eclipse-build
…infrastructure into eclipse-build
Nagaprasadvr
force-pushed
the
das-70-optimize-search-assets-query
branch
from
5ff6ccc to
6b9a079
Compare
|
linking this in https://github.com/rpcpool/digital-asset-rpc-infrastructure |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.