[EFM Recovery] Benchnet Testing

[jordanschalm]

Problem Definition

This issue enumerates test cases to manually test using a Benchnet instance, once the round-trip functionality of EFM Recovery is completed.

Test Cases - EFM Recovery

• Send recoverEpoch transaction twice
  It is possible that, due to a race condition and human error, a recoveryEpoch transaction is submitted which is accepted by the smart contract but rejected by the Protocol State. Specifically this can happen when the transaction is submitted near the boundary where a new EpochExtension is added, making the provided firstView invalid.

  We should be able to re-submit a new epochRecover transaction and enter the recovery epoch, while only paying rewards once.

• Test a recoverEpoch where, from the protocol state perspective, the RecoverEpoch counter is more than one greater than the epoch before EFM was triggered.
  This can be accomplished by performing several rounds of reward payouts prior to the recoverEpoch. This is no longer allowed in our data model, as epoch counters must be consecutive from the Protocol State POV.

• Test a recoverEpoch where new nodes have staked over the course of the epoch where EFM is triggered.

  We should be resilient to this case. In particular, newly staked nodes should not be included in the identity table of the RecoveryEpoch.

• Test a recoveryEpoch were we include include nodes in recovery transaction that were staked in the staking phase but never got included in the identity table since epoch has entered EFM.

• Test a recoveryEpoch where existing nodes have unstaked over the course of the epoch where EFM is triggered.

  We should be resilient to this case. In particular, unstaked CONSENSUS nodes must STILL BE included in the identity table of the RecoveryEpoch. All other unstaked nodes should not be included in the identity table of the RecoveryEpoch Test run

• Test a happy path scenario where network fails to complete DKG, enters EFM, runs a successful recoveryEpoch enters recovery epoch and successfully runs DKG and enters the epoch after.

  We should be able to operate normally after recovering epoch, test this scenario, especially if next DKG after recovery completes successfully. Test run

Since smart contract requires that all participants are staked and by the time of recovery potential participants have locked their stake then we should be able to successfully process this case

[jordanschalm]

First round of Benchnet testing, only covering entering and maintaining EFM with extensions is noted https://www.notion.so/flowfoundation/EFM-Recovery-Benchnet-Testing-Crescendo-Release-1bbd4eabe1ee41688b51ee7487c84822?pvs=4

[jordanschalm]

Starting second round of Benchnet testing, for complete feature. Will be posting testing notes to https://www.notion.so/flowfoundation/1351aee1232480fcb66bfb558e8cadd5?v=7e228fe9894945909608aef2f4ffafd0&pvs=4

[jordanschalm]

Added #6849 to capture Benchnet tests for the Protocol HCU upgrade process (rather than the EFM Recovery feature).

[durkmurder]

We also need to ensure that we can properly enter next epoch after the recovery, run a successful DKG and enter the epoch after the recovery one. We weren't able to test this case with integration tests.