Slow mining of pending transactions

[praetoriansentry]

Generally, if the pending queue has transactions in it, Erigon will rapidly create blocks and consume all of the transactions within the pending queue. E.g. often the blocks will be created with a 1s or even 0s block interval.

With the latest version, I'm seeing some different behavior where Erigon will have many valid transactions in the pending queue, but it mines empty blocks and it takes a very long time to process the transactions.

In the screenshot above, I've sent about 200 identical transactions. They're essentially all valid and pending. If I wait long enough they'll all be mined. But in the mean time Erigon is not producing blocks quickly and in fact it's mining empty blocks. These are the logs from Erigon that align with the time period shown in the screenshot:
erigon-slow-mining.log

To replicate the scenario above, I'm sending a bunch of transactions like this:

private_key="0x12d7de8621a77640c9241b2595ba78ce443d05e94090365ab3bb5e19df82c625"
rpc_url="$(kurtosis port print cdk cdk-erigon-node-001 rpc)"
eth_address="$(cast wallet address --private-key $private_key)"
nonce="$(cast nonce --rpc-url $rpc_url $eth_address)"

echo "$nonce"

for i in {1..200}; do
    echo $i
    cast send --legacy \
         --async \
         --nonce "$nonce" \
         --rpc-url "$rpc_url" \
         --private-key "$private_key" \
         --gas-limit 186657 \
         --create 5f5b60055a11156013576001810190506001565b50

    nonce="$((nonce+1))"
done

If I change the --gas-limit to be 373315 instead, the behavior is very different and a lot of blocks will be created right away. In this scenario, I'm running with Erigon 434376b and using the fork13 configuration:

kurtosis run --enclave cdk --args-file ./.github/tests/combinations/fork13-new-cdk-stack-cdk-validium.yml .

[praetoriansentry]

FYI @xavier-romero pointed out that this is likely related or the same issue as #1308. In both cases, transactions stay pending while we wait for the batch to close.

[hexoscott]

I'd imagine this is fixed now by #1417. If you enable zkevm.seal-batch-immediately-on-overflow it will close the batch if there are already transactions in the batch and an overflow is detected.

[Sharonbc01]

Review again in beta 6 @praetoriansentry @xavier-romero

[Sharonbc01]

Hi @praetoriansentry can this be tested using an image of zkEVM. Hoping we can close out this week if possible 🤞

[praetoriansentry]

It seems manageable now with this setting