Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - sync2: add sqlstore #6405

Closed
wants to merge 20 commits into from
Closed

Commits on Oct 23, 2024

  1. sync2: improve rangesync Receive semantics

    Given that after recent item sync is done (if it's needed at all), the
    range set reconciliation algorithm no longer depends on newly received
    item being added to the set, we can save memory by not adding the
    received items during reconciliation.
    
    During real sync, the received items will be sent to the respective
    handlers and after the corresponding data are fetched and validated,
    they will be added to the database, without the need to add them to
    cloned OrderedSets which are used to sync against particular peers.
    ivan4th committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    86b9591 View commit details
    Browse the repository at this point in the history
  2. sync2: implement multi-peer synchronization

    This adds multi-peer synchronization support.
    When the local set differs too much from the remote sets,
    "torrent-style" "split sync" is attempted which splits the set into
    subranges and syncs each sub-range against a separate peer.
    Otherwise, the full sync is done, syncing the whole set against
    each of the synchronization peers.
    Full sync is also done after each split sync run.
    The local set can be considered synchronized after the specified
    number of full syncs has happened.
    
    The approach is loosely based on [SREP: Out-Of-Band Sync of
    Transaction Pools for Large-Scale
    Blockchains](https://people.bu.edu/staro/2023-ICBC-Novak.pdf) paper by
    Novak Boškov, Sevval Simsek, Ari Trachtenberg, and David Starobinski.
    ivan4th committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    ef30f47 View commit details
    Browse the repository at this point in the history
  3. sync2: add sqlstore

    The `sqlstore` package provides simple sequence-based interface to the
    tables being synchronized. It is used by the FPTree data structure as
    the database layer, and doesn't do range fingerprinting by itself.
    
    `SyncedTable` and `SyncedTableSnapshot` provide methods that wrap the
    necessary SQL operations. `sql/expr` package was added to facilitate
    SQL generation.
    ivan4th committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    bb43161 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2024

  1. Configuration menu
    Copy the full SHA
    af95c7f View commit details
    Browse the repository at this point in the history
  2. sync2: doc update

    ivan4th committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    b062eaa View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. Configuration menu
    Copy the full SHA
    3bfaa3f View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. Configuration menu
    Copy the full SHA
    ef484c5 View commit details
    Browse the repository at this point in the history
  2. sync2: address comments

    ivan4th committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    0cf1678 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    287d76d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    51166fa View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2024

  1. Configuration menu
    Copy the full SHA
    c58d690 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    cf46587 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    79eb4db View commit details
    Browse the repository at this point in the history
  4. sync2: sqlstore: fix comment

    ivan4th committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    9442698 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. sync2: removed LRU cache

    It didn't add much to efficiency
    ivan4th committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    8a8c60c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9d6441f View commit details
    Browse the repository at this point in the history
  3. sync2: fix lint issue

    ivan4th committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    6424aca View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2024

  1. sync2: sqlstore: cache generated SQL

    Re-generating SQL statements every time is quick but increases GC
    pressure.
    ivan4th committed Nov 2, 2024
    Configuration menu
    Copy the full SHA
    79a31b7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    59f7db5 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2024

  1. Configuration menu
    Copy the full SHA
    3750cb7 View commit details
    Browse the repository at this point in the history