Increase the maximum of ancient slots allowed to be not combined

[dmakarov]

Problem

Large files containing ancient slots are expensive to update when the contained account data is being overwritten in more recent slots.

Summary of Changes

We increase the value of the tuning parameter max_ancient_slots to allow more ancient slots to remain in their individual storages instead of being combined into larger files. The smaller files are faster to update at the expense of the validator using more file handles.

[dmakarov]

I'm working on adding an option to set this parameter on the validator's command line.

[jeffwashington]

I suggest coupling this change with:
ANCIENT_APPEND_VEC_DEFAULT_OFFSET = 100_000

This keeps the overall 'ancient' range at 100k max, and the overall storage range at 432k, so we are at status quo relative today regarding # slots and # file handles.

[HaoranYi]

max file handle = 430K + 100K + 100K = 630K.
not too bad.
lgtm!

[jeffwashington]

max file handle = 430K + 100K + 100K = 630K. not too bad. lgtm!

I think 100k means move it into the 432k. I think -10k moves it past the end (so max of 432k - - 10k = 442k.

I'm proposing 432k - 100k + 100k = 432k total.

[HaoranYi]

Ah. I see.

So we will update this in the next PR?

agave/accounts-db/src/accounts_db.rs

Line 598 in 369d2cb

const ANCIENT_APPEND_VEC_DEFAULT_OFFSET: Option<i64> = Some(-10_000);

[jeffwashington]

Ah. I see.

So we will update this in the next PR?

agave/accounts-db/src/accounts_db.rs

Line 598 in 369d2cb

const ANCIENT_APPEND_VEC_DEFAULT_OFFSET: Option<i64> = Some(-10_000);

I wanted that done IN this pr:

#3205 (comment)

I guess I could've been more clear.

[dmakarov]

Ah. I see.

So we will update this in the next PR?

agave/accounts-db/src/accounts_db.rs

Line 598 in 369d2cb

const ANCIENT_APPEND_VEC_DEFAULT_OFFSET: Option<i64> = Some(-10_000);

in this one.

[jeffwashington]

i think this is + 100_000

[dmakarov]

shouldn't it be an offset to smaller slot numbers? if the offset is positive, the logic of several tests has to be adjusted at least, maybe some logic in the algorithms as well. let me figure this out.

[jeffwashington]

we definitely need to start a skipping rewrites version of this and it needs to run more than an epoch. Skipping rewrites is still opt-in and the current master behavior is unacceptable, so even if this isn't completely right, the risk is low at the moment.

[dmakarov]

maybe I should merge #3206, rebase this on new master, and then start a skipping rewrites validator? so that all tweaks are included...

[jeffwashington]

maybe I should merge #3206, rebase this on new master, and then start a skipping rewrites validator? so that all tweaks are included...

@dmakarov yes, I just tracked these down. 3206 is approved. I say merge it.

[dmakarov]

Is there something we're waiting for this PR? I think I addressed all the comments.

[HaoranYi]

lgtm

[brooksprumo]

Looks good to me. Happy to approve once #3266 is merged.

[dmakarov]

Looks good to me. Happy to approve once #3266 is merged.

it's merged.

[jeffwashington]

Is there something we're waiting for this PR? I think I addressed all the comments.

My testing found that we can get stuck when I enabled this change. We are not reducing alive_bytes when we encounter multi ref storages we can't pack yet.
We also aren't shrinking every ancient storage with dead bytes - just the ones < 90% alive.
I think both of those need to go in or we may find ourselves in a looming oom situation like I did where we fail to make progress.

[dmakarov]

Is there something we're waiting for this PR? I think I addressed all the comments.

My testing found that we can get stuck when I enabled this change. We are not reducing alive_bytes when we encounter multi ref storages we can't pack yet. We also aren't shrinking every ancient storage with dead bytes - just the ones < 90% alive. I think both of those need to go in or we may find ourselves in a looming oom situation like I did where we fail to make progress.

so this depends on #3251 and #3252 then.

[jeffwashington]

lgtm. dependencies are merged