Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch insert blocks #272

Merged
merged 7 commits into from
Oct 23, 2024
Merged

Batch insert blocks #272

merged 7 commits into from
Oct 23, 2024

Conversation

limemloh
Copy link
Contributor

@limemloh limemloh commented Oct 21, 2024 •
edited
Loading

Purpose

Reduce indexer catch-up time.

Benchmarked by running the indexer for exactly 10 minutes on a local set up, (i.e. indexer, database and concordium node (testnet) all running on the same machine) resetting the database in between:

  • Without batching it managed to index to block height 86811.
  • With block batching it managed to index to block height 124171.

Changes

  • When processing a batch, collect all the blocks and insert them as one query, reducing the chattiness with the database.
  • Spawn separate tasks for preprocessing and processing, allowing for these to be done in parallel.

Victor-N-Suadicani
Victor-N-Suadicani approved these changes Oct 21, 2024
View reviewed changes
Copy link
Contributor

@Victor-N-Suadicani Victor-N-Suadicani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few suggestions/questions.

backend-rust/src/indexer.rs Show resolved Hide resolved
backend-rust/src/indexer.rs Show resolved Hide resolved
backend-rust/src/indexer.rs Show resolved Hide resolved
backend-rust/src/indexer.rs Outdated Show resolved Hide resolved
backend-rust/src/indexer.rs Outdated Show resolved Hide resolved
backend-rust/src/indexer.rs Outdated Show resolved Hide resolved
DOBEN
DOBEN reviewed Oct 21, 2024
View reviewed changes
backend-rust/src/indexer.rs Outdated Show resolved Hide resolved
backend-rust/src/indexer.rs Outdated Show resolved Hide resolved
backend-rust/src/indexer.rs Outdated Show resolved Hide resolved
backend-rust/src/indexer.rs Outdated Show resolved Hide resolved
backend-rust/src/indexer.rs Outdated Show resolved Hide resolved
backend-rust/src/indexer.rs Outdated Show resolved Hide resolved
backend-rust/src/indexer.rs Outdated Show resolved Hide resolved
@limemloh limemloh requested a review from DOBEN October 21, 2024 18:51
@limemloh limemloh merged commit 3050f09 into rust-backend Oct 23, 2024
3 checks passed
@limemloh limemloh deleted the batch-blocks branch October 23, 2024 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Victor-N-Suadicani Victor-N-Suadicani Victor-N-Suadicani approved these changes

@DOBEN DOBEN DOBEN approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@limemloh @DOBEN @Victor-N-Suadicani