Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel synchronization #78

Merged
merged 98 commits into from
Oct 19, 2023
Merged

Parallel synchronization #78

merged 98 commits into from
Oct 19, 2023

Commits on Oct 6, 2023

  1. smaller channel for changes, using too much memory right now

    @jeromegn
    jeromegn committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    c5ab6dc View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2023

  1. track QUIC connection metrics, in aggregate

    @jeromegn
    jeromegn committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    47b5c30 View commit details
    Browse the repository at this point in the history

  2. easier to query these metrics if they have the same name but differen… 

    …t labels
    @jeromegn
    jeromegn committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    a61664e View commit details
    Browse the repository at this point in the history

  3. higher limit of max unidirectional QUIC streams

    @jeromegn
    jeromegn committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    4320746 View commit details
    Browse the repository at this point in the history

  4. limit number of concurrent syncs to 3

    @jeromegn
    jeromegn committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    f5d7220 View commit details
    Browse the repository at this point in the history

  5. pool unidirectional streams so we don't create nearly as many

    @jeromegn
    jeromegn committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    34ee8e8 View commit details
    Browse the repository at this point in the history

  6. don't pool unidirectional send streams

    @jeromegn
    jeromegn committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    62329e1 View commit details
    Browse the repository at this point in the history

  7. finish unidirectional streams when sending data

    @jeromegn
    jeromegn committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    c247b3c View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2023

  1. emit tokio runtime metrics

    @jeromegn
    jeromegn committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    53fd61b View commit details
    Browse the repository at this point in the history

  2. track cwdn per addr

    @jeromegn
    jeromegn committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    6a56f11 View commit details
    Browse the repository at this point in the history

  3. track congestion events and blackhols per addr

    @jeromegn
    jeromegn committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    efddae3 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. retry opening a unidirectional stream once if it failed when trying t… 

    …o send a unidirectional payload
    @jeromegn
    jeromegn committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    a5747c6 View commit details
    Browse the repository at this point in the history

  2. split booked versions in their state (cleared, current, partial)

    @jeromegn
    jeromegn committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    5f49d56 View commit details
    Browse the repository at this point in the history

  3. first part of syncing more efficiently between nodes

    @jeromegn
    jeromegn committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    06bf45f View commit details
    Browse the repository at this point in the history

  4. fixed sync mistake

    @jeromegn
    jeromegn committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    cabceda View commit details
    Browse the repository at this point in the history

  5. move comment

    @jeromegn
    jeromegn committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    76501e2 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2023

  1. wip parallel sync

    @jeromegn
    jeromegn committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    d081764 View commit details
    Browse the repository at this point in the history

  2. hopefully massive improvements on sync speed, especially from scratch

    @jeromegn
    jeromegn committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    f0bb3ee View commit details
    Browse the repository at this point in the history

  3. patch upgrade quinn-proto to use less memory

    @jeromegn
    jeromegn committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    ed895a6 View commit details
    Browse the repository at this point in the history

  4. observe member syncing

    @jeromegn
    jeromegn committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    c173a7c View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2023

  1. something is weird here, syncing should be a lot faster

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    59300e5 View commit details
    Browse the repository at this point in the history

  2. hopefully fix tests, add cluster inspection commands

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    37a7a54 View commit details
    Browse the repository at this point in the history

  3. measure how many sent sync requests

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    52762b1 View commit details
    Browse the repository at this point in the history

  4. send more often

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    3438e90 View commit details
    Browse the repository at this point in the history

  5. process completed empties in much smaller batches

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    359939e View commit details
    Browse the repository at this point in the history

  6. log a bit more

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    49ae55d View commit details
    Browse the repository at this point in the history

  7. send much smaller chunks

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    e282a72 View commit details
    Browse the repository at this point in the history

  8. possibly faster cleared empties processing

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    9a714bb View commit details
    Browse the repository at this point in the history

  9. need to acquire an async read lock here

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    960afd0 View commit details
    Browse the repository at this point in the history

  10. don't timeout on receive

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    efbf224 View commit details
    Browse the repository at this point in the history

  11. send sync requests in bigger chunks

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    88396ae View commit details
    Browse the repository at this point in the history

  12. better metrics for changes received and lookout for the send stream s… 

    …topping
    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    1fd747a View commit details
    Browse the repository at this point in the history

  13. try more servers

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    27a4695 View commit details
    Browse the repository at this point in the history

  14. try to aggregate on the server-side too

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    4f7f2f6 View commit details
    Browse the repository at this point in the history

  15. process smaller chunks

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    4292492 View commit details
    Browse the repository at this point in the history

  16. reduce number of locks being acquired

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    da662cd View commit details
    Browse the repository at this point in the history

  17. process each chunk of sync requests concurrently

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    6c5db98 View commit details
    Browse the repository at this point in the history

  18. bring back chunking for performance reasons

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    52554ed View commit details
    Browse the repository at this point in the history

  19. different concurrency attempt

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    c2c95ca View commit details
    Browse the repository at this point in the history

  20. properly break out of loop

    @jeromegn
    jeromegn committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    2a013cf View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2023

  1. don't break until we've read everything

    @jeromegn
    jeromegn committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    b8a368f View commit details
    Browse the repository at this point in the history

  2. send success to the response ends

    @jeromegn
    jeromegn committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    9a2504e View commit details
    Browse the repository at this point in the history

  3. log which actor_id returned an error

    @jeromegn
    jeromegn committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    f35501a View commit details
    Browse the repository at this point in the history

  4. allow a few more changes in the channels

    @jeromegn
    jeromegn committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    cf84a44 View commit details
    Browse the repository at this point in the history

  5. send smaller chunks

    @jeromegn
    jeromegn committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    8ad87e7 View commit details
    Browse the repository at this point in the history

  6. try to block less when processing empties

    @jeromegn
    jeromegn committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    cefbf8a View commit details
    Browse the repository at this point in the history

  7. better qualify lock name for booked writer when processing multiple v… 

    …ersions, don't process empties constantly
    @jeromegn
    jeromegn committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    23cfda8 View commit details
    Browse the repository at this point in the history

  8. let's try applying more changes now that empties are out of the way

    @jeromegn
    jeromegn committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    1d8f2db View commit details
    Browse the repository at this point in the history

  9. don't count empties when determining when to process multiple changes

    @jeromegn
    jeromegn committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    a379ae5 View commit details
    Browse the repository at this point in the history

  10. log how many changes were inserted and in how much time

    @jeromegn
    jeromegn committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    6921c0b View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2023

  1. when clearing detected-cleared db versions, do it lazily as part of t… 

    …he normal process, bigger batches for inserts, measure more times
    @jeromegn
    jeromegn committed Oct 15, 2023
    Configuration menu
    Copy the full SHA
    cb411b4 View commit details
    Browse the repository at this point in the history

  2. manually declare actor a down if its timestamp is lower than previous… 

    …ly recorded
    @jeromegn
    jeromegn committed Oct 15, 2023
    Configuration menu
    Copy the full SHA
    dd2b81c View commit details
    Browse the repository at this point in the history

  3. use duration instead of value of u64 or timestamp

    @jeromegn
    jeromegn committed Oct 15, 2023
    Configuration menu
    Copy the full SHA
    a77deb5 View commit details
    Browse the repository at this point in the history

  4. log less

    @jeromegn
    jeromegn committed Oct 15, 2023
    Configuration menu
    Copy the full SHA
    2bc13c1 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2023

  1. attempt to add open-telemetry tracing

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    59477ec View commit details
    Browse the repository at this point in the history

  2. instrument a ton more

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    9e264fd View commit details
    Browse the repository at this point in the history

  3. unclog connection locks

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    b42f4b9 View commit details
    Browse the repository at this point in the history

  4. reduce lock times

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    e7de255 View commit details
    Browse the repository at this point in the history

  5. don't include insane span list in JSON

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    2e246e4 View commit details
    Browse the repository at this point in the history

  6. don't have to clone a full connection, just get its stats

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    266b708 View commit details
    Browse the repository at this point in the history

  7. reduce level of read/write bookie locks so we don't spam otel

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    a03f841 View commit details
    Browse the repository at this point in the history

  8. still trying to fix otel

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    9e2a783 View commit details
    Browse the repository at this point in the history

  9. set send_datagram to debug log level

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    050412b View commit details
    Browse the repository at this point in the history

  10. maybe this helps

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    98d248f View commit details
    Browse the repository at this point in the history

  11. make a parent span somehow to better track process_multiple_changes

    @jeromegn
    jeromegn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    29f3f71 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. remove instrumentation to see if it helps

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    0102bb7 View commit details
    Browse the repository at this point in the history

  2. using a span across awaits can lead to weird stuff

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    c56b57d View commit details
    Browse the repository at this point in the history

  3. don't know why but this needs to be an async rwlock

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    d51ac81 View commit details
    Browse the repository at this point in the history

  4. more instrumentation fiddling

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    50bfbcb View commit details
    Browse the repository at this point in the history

  5. more instrumentation, including result errors and such

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    bc86463 View commit details
    Browse the repository at this point in the history

  6. remove span across awaits

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    a2d8f0e View commit details
    Browse the repository at this point in the history

  7. propagate trace context when starting sync

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    2a06e9a View commit details
    Browse the repository at this point in the history

  8. more instrumentation for sync process, both ways

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    05e6cd5 View commit details
    Browse the repository at this point in the history

  9. less instrumentation for stuff that happens too often

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    bf330ee View commit details
    Browse the repository at this point in the history

  10. try to get rid of more traces

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    054e241 View commit details
    Browse the repository at this point in the history

  11. log needs len

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    01875ac View commit details
    Browse the repository at this point in the history

  12. don't block when sending into empties channel

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    15cbe87 View commit details
    Browse the repository at this point in the history

  13. seqs start at 0

    @jeromegn
    jeromegn committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    79091b8 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2023

  1. add a timeout to quic connect

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    c19d2f2 View commit details
    Browse the repository at this point in the history

  2. warn if sending too large datagram, log its size

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    7da737a View commit details
    Browse the repository at this point in the history

  3. also log max datagram size

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    966aa23 View commit details
    Browse the repository at this point in the history

  4. set a smaller max_packet_size for foca, appears we are limited by pat… 

    …h mtu
    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    66fd461 View commit details
    Browse the repository at this point in the history

  5. don't require locking the bookie to get at the registry

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    2de4bf4 View commit details
    Browse the repository at this point in the history

  6. fix a deadlock

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    fb2248e View commit details
    Browse the repository at this point in the history

  7. allow far more unidirectional streams

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    a4b38c9 View commit details
    Browse the repository at this point in the history

  8. try to simplify reading from unidirectional stream

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    e8f631b View commit details
    Browse the repository at this point in the history

  9. use multiple endpoints to parallelize sending data

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    35d0e19 View commit details
    Browse the repository at this point in the history

  10. use seahash instead of manually modulo-ing to figure out endpoint index

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    3abf6fc View commit details
    Browse the repository at this point in the history

  11. restore corro_peer_stream_bytes_recv_total metric

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    08d5493 View commit details
    Browse the repository at this point in the history

  12. reduce number of max concurrent uni streams

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    db1bca1 View commit details
    Browse the repository at this point in the history

  13. record gossip recv lag

    @jeromegn
    jeromegn committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    94adf28 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2023

  1. don't send empty trace states

    @jeromegn
    jeromegn committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    444c017 View commit details
    Browse the repository at this point in the history

  2. more prometheus histogram buckets for more precision

    @jeromegn
    jeromegn committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    143ad89 View commit details
    Browse the repository at this point in the history

  3. fix panic, somehow it received a 0-value head version in the sync gen… 

    …erated from another actor, added a log to investigate next time it happens
    @jeromegn
    jeromegn committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    f94b0fa View commit details
    Browse the repository at this point in the history

  4. update CHANGELOG

    @jeromegn
    jeromegn committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    04586cc View commit details
    Browse the repository at this point in the history

  5. upgrade mdbook-admonish

    @jeromegn
    jeromegn committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    c667075 View commit details
    Browse the repository at this point in the history

  6. probably fix rustflags on CI

    @jeromegn
    jeromegn committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    ba4b29e View commit details
    Browse the repository at this point in the history

  7. try sccache instead fo rust-cache

    @jeromegn
    jeromegn committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    4ca321f View commit details
    Browse the repository at this point in the history