Aggressively shrink ancient storages when shrink isn't too busy. #2946

dmakarov · 2024-09-16T21:00:47Z

Problem

Ancient packing when skipping rewrites has some non-ideal behavior.
It can sometimes be true that an ancient storage might never meet the 90%(?) threshold for shrinking. However, every dead account that an ancient storage keeps present causes the account to remain in the index in memory and starts a chain reaction of other accounts, such as zero lamport accounts, that must be kept alive.

Summary of Changes

Add another slot for shrinking when the number of shrink candidate slots is too small (less than 10). The additional slot's storage has the largest number of dead bytes. This aggressively shrinks ancient storages, even when they are below the normal threshold. This allows the system to keep itself towards the ideal of storing each non-zero account once and having no zero lamport accounts.

reworked #2849

jeffwashington · 2024-09-27T19:20:37Z

@dmakarov, where did you end up on this? We can move this forward monday. Hopefully you have a machine running this?

dmakarov · 2024-09-27T19:24:39Z

@dmakarov, where did you end up on this? We can move this forward monday. Hopefully you have a machine running this?

I had it running on my dev machine for a few days. Now I added stat counters and will restart it with the new stats. I’d like to experiment a bit more with this.

accounts-db/src/ancient_append_vecs.rs

jeffwashington · 2024-09-30T23:04:28Z

please add a test that shows we add an ancient storage to shrink. May be helpful to refactor the shrink fn so that it calculates the storages to shrink in a separate fn so we can just check the output of that fn. Or, you can do a more full test which actually verifies the capacity is what you expect after running shrink. There should be tests that do this similarly already. Look for tests that call shrink_candidate_slots

dmakarov · 2024-10-01T15:13:30Z

please add a test that shows we add an ancient storage to shrink.

yes, I'm working on it.

jeffwashington · 2024-10-01T15:37:26Z

Here's what we get with this change and the rest of the ancient packing tweaks:

purple and blue are the prior code which maybe added up to 10 ancient storages to pack. Now we're consistently adding 1. Are we making enough progress? I'm not sure. Seems like it may take 4 days to shrink each ancient storage once.

jeffwashington · 2024-10-01T15:44:13Z

jw10 is the machine I just added this change onto (with other ancient packing changes.
This pr is definitely not wrong. It may be insufficient. A metric to watch is # in-mem index entries. We'll have in-mem idx entries if we have dead accounts accumulating in ancient storages. Dead accounts means we have refcount > 1, thus they will remain in memory.

Another metric is # zero lamport ancient accounts. This will be higher if we aren't aggressively cleaning ancient storages.

jeffwashington · 2024-10-01T15:49:14Z

this pr should have zero effect unless skipping rewrites is enabled by cli.

accounts-db/src/accounts_db.rs

HaoranYi

The idea in the PR looks good to me.
I don't see too much downside for adding one ancient to shrink when we are not busy.

accounts-db/src/accounts_db.rs

brooksprumo · 2024-10-01T19:58:56Z

accounts-db/src/accounts_db.rs

+                        && *capacity == store.capacity()
+                        && Self::is_candidate_for_shrink(self, &store)
+                    {
+                        *capacity = 0;


Can we not overload the u64, and instead create an enum to indicate if this storage is pre or post shrunk?

I don't know. Isn't capacity checked for being 0 in other logic, so that if we add an enum we still have to set capacity to 0 here for other code to work correctly?

we are overloading the 0 here. Yes. We could do an enum and it would be much more clear what we're trying to do.
{AlreadyShrunk, CanBeShrunk(capacity: u64)}

could also just remove the element from the vec, but that could be expsneive. I was assuming marking it as 'already shrunk' would be sufficient. maybe none of htis is necessary because we'll see that the new capacity doesn't match the old capacity and skip it anyway... Then we don't need to iter mut at all and we can just iter. That seems simplest of all and we already have to handle that case anyway.

This does cause us to look up way more storages.
An oldie but goodie: https://en.wikichip.org/wiki/schlemiel_the_painter%27s_algorithm

What is the suggested change? Not to change capacity?

enum is fine with me. iterating. alternatively, vec sorted in reverse and you pop the last one off the end and reduce the count. This would not require a re-allocation and would avoid revisiting ancient storages we already previously shrunk.

accounts-db/src/accounts_db.rs

accounts-db/src/ancient_append_vecs.rs

dmakarov · 2024-10-02T14:05:24Z

it looks like i need to rebase to fix the vulnerability check errors.

dmakarov · 2024-10-02T14:15:03Z

rebased to resolve conflicts. I'm still working on unit tests. When ready, I'll renew review requests. Thanks.

jeffwashington · 2024-10-15T14:27:24Z

accounts-db/src/ancient_append_vecs.rs

+            &mut ancient_slot_infos.best_slots_to_shrink,
+        );
+        // Reverse the vector so that the elements with the largest
+        // dead bytes are poped first when used to extend the


jeffwashington · 2024-10-15T14:29:45Z

accounts-db/src/ancient_append_vecs.rs

+        // Reverse the vector so that the elements with the largest
+        // dead bytes are poped first when used to extend the
+        // shrinking candidates.
+        self.best_ancient_slots_to_shrink.write().unwrap().reverse();


probably reverse them while it is still local, before swapping

jeffwashington · 2024-10-15T14:30:29Z

accounts-db/src/accounts_db.rs

@@ -1463,6 +1468,11 @@ pub struct AccountsDb {
    /// Flag to indicate if the experimental accounts lattice hash is enabled.
    /// (For R&D only; a feature-gate also exists to turn this on and make it a part of consensus.)
    pub is_experimental_accumulator_hash_enabled: AtomicBool,
+
+    /// These are the ancient storages that could be valuable to shrink.
+    /// sorted by largest dead bytes to smallest


i think they are sorted now smallest dead bytes to largest? I don't see where we are getting that sort order just from the diffs here and I don't quite remember.

I just added sorting in another commit. I had to add a field to the tuple, so that we actually sort the elements by the amount of dead bytes.

jeffwashington · 2024-10-15T14:47:30Z

accounts-db/src/accounts_db.rs

+        // assumed to be in reverse order.
+        if shrink_slots.len() < SHRINK_INSERT_ANCIENT_THRESHOLD {
+            let mut ancients = self.best_ancient_slots_to_shrink.write().unwrap();
+            while let Some((slot, capacity)) = ancients.pop() {


the pop is beautiful compared to my hacky original impl!

jeffwashington · 2024-10-15T15:15:11Z

accounts-db/src/ancient_append_vecs.rs

@@ -182,7 +183,8 @@ impl AncientSlotInfos {
        self.best_slots_to_shrink = Vec::with_capacity(self.shrink_indexes.len());
        for info_index in &self.shrink_indexes {


@dmakarov sorry to go in circles... reverse is probably right and simplest. If you look at sort_shrink_indexes_by_bytes_saved, I think we are already iterating in most bytes to save to least. So, reversing best_slots_to_shrink will be sorted correctly without the addition of a new field.

I think it's sorted on capacity not on the amount of dead bytes, though. Isn't it?

jeffwashington · 2024-10-15T15:30:09Z

accounts-db/src/ancient_append_vecs.rs

        // dead bytes are popped first when used to extend the
        // shrinking candidates.
-        self.best_slots_to_shrink.sort_by(|a, b| b.2.cmp(&a.2));
+        self.best_slots_to_shrink.reverse();


Should we reverse the Vec, or use a VecDeque and pop_front instead?

it's an option. how strongly do you feel about it for this pr?

Not strong.

i'll change it to a deque in a follow-up pr.

jeffwashington

lgtm

dmakarov · 2024-10-15T15:43:17Z

lgtm

sorry. had to update a comment.

brooksprumo

jeffwashington

lgtm

HaoranYi

lgtm

…a-xyz#2946) * Tweak ancient packing algorithm * Minor change * Feedback * Remove redundancy * Correction * Revert correction * Loop * Add test * Fix clippy * Comments * Comment * Comments * Pop ancients * Revert * Checks * Move reverse * Typo * Popped * Sort * Format * Revert sort, back to reverse * Fix comment

dmakarov force-pushed the packing branch 2 times, most recently from 9a2729a to 838e063 Compare September 16, 2024 21:44

dmakarov marked this pull request as ready for review September 17, 2024 01:09

dmakarov requested review from jeffwashington and brooksprumo September 17, 2024 12:58

jeffwashington requested a review from HaoranYi September 18, 2024 02:05

dmakarov force-pushed the packing branch 2 times, most recently from 4d869c2 to 003ba1f Compare September 27, 2024 18:12

dmakarov force-pushed the packing branch 2 times, most recently from b18e934 to 1bfd6d0 Compare September 27, 2024 20:59

jeffwashington reviewed Sep 30, 2024

View reviewed changes

accounts-db/src/ancient_append_vecs.rs Outdated Show resolved Hide resolved

dmakarov force-pushed the packing branch from 1bfd6d0 to 7e82fe9 Compare September 30, 2024 22:50

jeffwashington changed the title ~~Tweak ancient packing algorithm~~ Aggressively shrink ancient storages when shrink isn't too busy. Oct 1, 2024

jeffwashington self-requested a review October 1, 2024 15:49

jeffwashington reviewed Oct 1, 2024

View reviewed changes

accounts-db/src/accounts_db.rs Show resolved Hide resolved

HaoranYi reviewed Oct 1, 2024

View reviewed changes

accounts-db/src/accounts_db.rs Outdated Show resolved Hide resolved

HaoranYi previously approved these changes Oct 1, 2024

View reviewed changes

dmakarov dismissed HaoranYi’s stale review via 9d9c67b October 1, 2024 17:33

brooksprumo reviewed Oct 1, 2024

View reviewed changes

HaoranYi reviewed Oct 1, 2024

View reviewed changes

accounts-db/src/ancient_append_vecs.rs Show resolved Hide resolved

dmakarov force-pushed the packing branch from 6a379a4 to a36616f Compare October 2, 2024 14:13

dmakarov requested a review from brooksprumo October 14, 2024 21:40

dmakarov added 4 commits October 15, 2024 08:28

Revert

e253a42

Checks

7c2f5f2

Move reverse

fa967e2

Typo

2c5b3c1

jeffwashington reviewed Oct 15, 2024

View reviewed changes

Popped

dd5f265

dmakarov requested a review from jeffwashington October 15, 2024 14:29

jeffwashington reviewed Oct 15, 2024

View reviewed changes

jeffwashington self-requested a review October 15, 2024 14:46

jeffwashington reviewed Oct 15, 2024

View reviewed changes

dmakarov added 2 commits October 15, 2024 11:11

Sort

61c15b7

Format

af85db2

dmakarov requested a review from jeffwashington October 15, 2024 15:14

jeffwashington reviewed Oct 15, 2024

View reviewed changes

Revert sort, back to reverse

2948f80

dmakarov requested a review from jeffwashington October 15, 2024 15:27

jeffwashington reviewed Oct 15, 2024

View reviewed changes

jeffwashington previously approved these changes Oct 15, 2024

View reviewed changes

Fix comment

c72217e

dmakarov dismissed jeffwashington’s stale review via c72217e October 15, 2024 15:42

dmakarov requested a review from jeffwashington October 15, 2024 15:53

brooksprumo approved these changes Oct 15, 2024

View reviewed changes

jeffwashington approved these changes Oct 15, 2024

View reviewed changes

dmakarov merged commit 1e800b1 into anza-xyz:master Oct 15, 2024
40 checks passed

HaoranYi reviewed Oct 15, 2024

View reviewed changes

dmakarov deleted the packing branch October 15, 2024 20:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Aggressively shrink ancient storages when shrink isn't too busy. #2946

Aggressively shrink ancient storages when shrink isn't too busy. #2946

dmakarov commented Sep 16, 2024 •

edited by jeffwashington

Loading

jeffwashington commented Sep 27, 2024

dmakarov commented Sep 27, 2024

jeffwashington commented Sep 30, 2024

dmakarov commented Oct 1, 2024

jeffwashington commented Oct 1, 2024

jeffwashington commented Oct 1, 2024 •

edited

Loading

jeffwashington commented Oct 1, 2024

HaoranYi left a comment

brooksprumo Oct 1, 2024

dmakarov Oct 1, 2024

jeffwashington Oct 1, 2024

jeffwashington Oct 1, 2024 •

edited

Loading

dmakarov Oct 1, 2024

jeffwashington Oct 7, 2024

dmakarov commented Oct 2, 2024

dmakarov commented Oct 2, 2024

jeffwashington Oct 15, 2024

jeffwashington Oct 15, 2024

jeffwashington Oct 15, 2024

dmakarov Oct 15, 2024

jeffwashington Oct 15, 2024

jeffwashington Oct 15, 2024

dmakarov Oct 15, 2024

jeffwashington Oct 15, 2024

brooksprumo Oct 15, 2024

dmakarov Oct 15, 2024

brooksprumo Oct 15, 2024

dmakarov Oct 15, 2024

jeffwashington left a comment

dmakarov commented Oct 15, 2024

brooksprumo left a comment

jeffwashington left a comment

HaoranYi left a comment

		@@ -182,7 +183,8 @@ impl AncientSlotInfos {
		self.best_slots_to_shrink = Vec::with_capacity(self.shrink_indexes.len());
		for info_index in &self.shrink_indexes {

Aggressively shrink ancient storages when shrink isn't too busy. #2946

Aggressively shrink ancient storages when shrink isn't too busy. #2946

Conversation

dmakarov commented Sep 16, 2024 • edited by jeffwashington Loading

Problem

Summary of Changes

jeffwashington commented Sep 27, 2024

dmakarov commented Sep 27, 2024

jeffwashington commented Sep 30, 2024

dmakarov commented Oct 1, 2024

jeffwashington commented Oct 1, 2024

jeffwashington commented Oct 1, 2024 • edited Loading

jeffwashington commented Oct 1, 2024

HaoranYi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffwashington Oct 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmakarov commented Oct 2, 2024

dmakarov commented Oct 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffwashington left a comment

Choose a reason for hiding this comment

dmakarov commented Oct 15, 2024

brooksprumo left a comment

Choose a reason for hiding this comment

jeffwashington left a comment

Choose a reason for hiding this comment

HaoranYi left a comment

Choose a reason for hiding this comment

dmakarov commented Sep 16, 2024 •

edited by jeffwashington

Loading

jeffwashington commented Oct 1, 2024 •

edited

Loading

jeffwashington Oct 1, 2024 •

edited

Loading