don't try to ancient pack already large storages #548
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Account state continues to grow.
We'd like to turn off rent exempt rewrites.
At that point, we accumulate old append vecs.
We append or pack those into 'ancient' append vecs.
The packing algorithm can go off the rails in certain testing scenarios.
Note that the ancient algorithm is currently 'append'. This code does not affect 'append' at all.
The packing algorithm is the future. Packing is enabled with --accounts-db-create-ancient-storage-packed
Summary of Changes
Apply some limits to what the packing algorithm will attempt.
In this case, we skip trying to pack storages that are not shrink candidates but are already close to ideal size.
This limit is a stop gap to prevent machines from running oom by trying to combine too many and too large append vecs in one pass (by not flushing the write cache most likely or by loading pubkeys into memory for billions of accounts).
Fixes #