Respect grace ticks

[bw-solana]

Problem

There's a bug where we will not respect grace ticks (currently set to 2 slots worth) because once we tick through 1 slot worth of ticks, we think we will now be building off our own block (or at least have produced a block) and don't need to respect grace ticks.

Summary of Changes

• Instead of deriving the previous slot from ticks and using the leader cache to extract the pubkey, just grab it from the start bank. Note that this is a subtle algorithm change
• Update test_reached_leader_tick to cover all the possible paths.

Notes

1. I'm hoping this PR can be a discussion starter. One potential problem I see is that it's possible we already produced (or partially produced) our first block but have now switched the fork we're building off of. This would causes us to respect grace ticks even if we don't really have to. I think this is an obscure case with relatively benign consequences, but I want to make sure this side effect isn't ignored.
2. There's still work to do... by respecting grace ticks, the current leader is potentially losing slots. This is because the next_poh_slot computed in reached_leader_slot is based on the current tick_height

[jstarry]

Nice, this was a straightforward fix!

[jstarry]

changes look good but there's a failing poh related test

[AshwinSekar]

I'm in favor of this change.

One potential problem I see is that it's possible we already produced (or partially produced) our first block but have now switched the fork we're building off of. This would causes us to respect grace ticks even if we don't really have to.

As a follow up I think we should keep track of whether we have already waited for grace ticks / produced any shreds for our current leader slots in order to skip waiting again.

This is because the next_poh_slot computed in reached_leader_slot is based on the current tick_height

I am also in favor of always producing the earliest slot we can once we reach grace ticks. Otherwise it seems like a slow previous leader can always cause you to skip half your slots.

Also it might be too early to call without some more data, but it seems like turning delay_leader_block_for_pending_fork on as the default is beneficial. Of the 7 slots we saw on mainnet recently that fit this category:

279650852
279589292
279773684
280090288
280260040
280519864
280352012

All were skipped with the flag off

[bw-solana]

changes look good but there's a failing poh related test

Had to update this test to pass in the pubkey from the PoH start bank so it would match, conclude that previous slot is ours, ignore grace ticks, and thus return we've reached our leader slot.