Avoid combining older slots with newer ones in ancient shrinking #3189
Conversation
dmakarov
force-pushed
the
shrink
branch
6 times, most recently
from
e873ea5 to
59de09f
Compare
dmakarov
changed the title
|
If the tuning parameter |
Yes. But we are still finding a long term equilibrium. I THINK we will end up ripping out this shrinking selection behavior. But, in the meantime, it could be very convenient to have this already present. |
i've been testing this for many weeks. It does what we want. We accumulate old, cold storages and newer things are combined together. We may be shrinking too much (too many bytes of i/o and index churn for too little savings), but that is tuning to the shrinking algorithm we can do. This change stops ancient packing from also shrinking older storages. When shrink is idle, it shrinks the most productive ancient storage with dead bytes. |
I started a validator with |
dmakarov
dismissed stale reviews from brooksprumo and jeffwashington
via
ae1131a
There was a problem hiding this comment.
Seems like it is good to remove the byte-saving heuristic from ancient pack.
Naturally, this heuristic should be owned by shrinking.
We have too many heuristics considered in ancient packing, - by higher slots,
by smaller size and by byte-savings.
Too many heuristics make reasoning hard. Those heuristics may also be conflict
with each other as described in the pr description.
Now leaving byte-saving bytes out of the picture of ancient packing and let
shrinking deal with byte-savings seems to be a good decoupling and make the
overall reasoning and tuning simpler.
lgtm
| @@ -3786,7 +3788,7 @@ pub mod tests { | |||
| // only allow 10k slots old enough to be ancient | |||
| max_ancient_slots: 10_000, | |||
| // re-combine/shrink 55% of the data savings this pass | |||
dmakarov
dismissed stale reviews from brooksprumo and jeffwashington
via
3bb5af2
…a-xyz#3189) * Avoid combining older slots with newer ones in ancient shrinking * Comment * Comment
Problem
shrink_ancient_slots bundles ancient storages together without considering the relative age of slots for which they contain account data. The combined storage can contain account data for very old slots along with the data for more recent slots, thus making it more likely that the combined storage will have to be updated soon. More uniform storages w.r.t. to the age of slots they contain would less likely have to be updated when account data is overwritten in newer slots.
Summary of Changes
This change tweaks a tuning parameter that determines which ancient slots are eligible for shrinking, and, in the process, for combining into a larger storage. Setting the
percent_of_alive_shrunk_dataparameter to 0, we effectively disable selecting old ancient slots for shrinking based only on their contribution of alive bytes to the combined storage alive bytes. These slots can still be picked up for shrinking whenshrink_candidate_slotsis called, but their storages won't be combined with the storages containing newer slots account data unless these older storages become small.