Improve unified scheduler pipelining by chunking

[ryoqun]

Problem

Unlike blockstore-processor, unified scheduler can start processing transaction async as soon as it is fed with transactions. However, entries are currently fed in one large sweep when both catching-after-full-repair and normal replaying stage (the notorious 100ms):

agave/core/src/replay_stage.rs

Line 1149 in 34e9932

let timer = Duration::from_millis(100);

Summary of Changes

Introduce chunked entry load api in Blockstore and use it only for unified scheduler. Hence, subsequent deshredding can overlap in time with already-submitted unified scheduler processing, improving pipeline efficiency.

Lastly, note that this is optimization only applicable to block verification. block production by unified scheduler won't benefit at all. (however, some measurable gain for block verification, hence a this pr)

before

ledger processed in 13 seconds, 977 ms
ledger processed in 13 seconds, 777 ms
ledger processed in 13 seconds, 905 ms

after

maybe 4-5% gain

ledger processed in 13 seconds, 209 ms
ledger processed in 13 seconds, 276 ms
ledger processed in 13 seconds, 207 ms

Extracted from: #2325

[ryoqun]

write tests?

[ryoqun]

best viewed in hide-whitespace mode diff. :)

[ryoqun]

i wonder how much overhead it incurs here instead of one-giant invocation of .multi_get_bytes().

[ryoqun]

After #2881, the bench crossed the 13-sec wall:

ledger processed in 12 seconds, 920 ms
ledger processed in 12 seconds, 935 ms
ledger processed in 13 seconds, 77 ms

for reference here's recent blockstore-processor numbers:

ledger processed in 27 seconds, 124 ms
ledger processed in 26 seconds, 902 ms
ledger processed in 26 seconds, 997 ms

[ryoqun]

note to self: this pr is kind of stale.. currently it's known this chunked behavior adversely affects entry verification rayon thread group.