Hardhat reports inconsistent gas usages when running in the fork mode

[polka125]

Version of Hardhat

2.22.15

What happened?

What Goes Wrong

Hardhat reports inconsistent gas usage when running in fork mode. Different runs may result in different gas usages on the same set of transactions in the same order, in the state forked from the same block.

A potentially related issue has been reported before.

My Setup

Hardhat Server Process

I use Hardhat as a local JSON_RPC. Here is my hardhat.config.js:

module.exports = {
    solidity: "0.8.25",
    networks:{
      hardhat:{
        throwOnTransactionFailures: false,
        throwOnCallFailures: false,
        chainId: 1,
        mining: {
          mempool: {
            order: "fifo"
          }
        },
        accounts: {
          count: 2
        }
      }
    }
  };
  

Payloads

I have a paylods.json file containing a set of queries. The detailed description of the queries structure is deferred to the further details section. The payloads.json is hosted here or find attached payloads.json

Payloads Execution

The payloads are submitted via POST request using python sctipt submit_payloads.py, which does the following;

For iteration 1 to MAX_RUNS

1. forks the blockchain using FORK_PAYLOAD payload

2. Submit each payload from the payloads.json file and logs the responses. The return status is ensured to be 200

3. logs the responses to the output_{run}.json file

4. prints the hashes of the logs upon exit to the screen

The full code is as below. Note that in order to run, you need to replace JSON_RPC_TOKEN with a token, I use infura api:

import json
import requests
import time
import os
import hashlib

JSON_RPC_TOKEN = "YOUR REMOTE RPC TOKEN"


# Tunable parameters
HARDHAT_PORT = 1234
MAX_RUNS = 200
COOLDOWN_SECONDS = 1
PAYLOADS = "payloads.json"
LOG_DIR = "logs"

HEADERS = {
    'Content-Type': 'application/json'
}
FORK_PAYLOAD = {
    "jsonrpc": "2.0",
    "method": "hardhat_reset",
    "params": [
        {
            "forking": {
                "jsonRpcUrl": JSON_RPC_TOKEN,
                "blockNumber": 20845407
            }
        }
    ],
    "id": 1
}
REQUEST_BLOCK_INFO_PAYLOAD = {
    "jsonrpc": "2.0",
    "method": "eth_getBlockByNumber",
    "params": ["latest", False],
    "id": 1
}


def submit_payload(payload):
    response = requests.request("POST", f"http://127.0.0.1:{HARDHAT_PORT}", headers=HEADERS, data=json.dumps(payload))  
    assert response.status_code == 200
    return json.loads(response.text)


def main(run):
    # first fork the mainnet
    submit_payload(FORK_PAYLOAD)

    # then read payloads from a file
    with open(PAYLOADS, 'r') as f:
        payloads = json.load(f)
    
    # submit each payload and log the response
    with open(f"{LOG_DIR}/output_{str(run).zfill(3)}.json", 'w') as f:
        for i, payload in enumerate(payloads):
            response = submit_payload(payload)
            
            # log the response
            f.write(json.dumps(response).replace("\n", ""))
            f.write("\n")

    # request block info
    response = submit_payload(REQUEST_BLOCK_INFO_PAYLOAD)

    # return the hash of the block
    return response


if __name__ == "__main__":
    if not os.path.exists(LOG_DIR):
        os.makedirs(LOG_DIR)

    for i in range(MAX_RUNS):
        main(i)
        time.sleep(1)


    log_files = os.listdir(LOG_DIR)
    log_files = [fname for fname in log_files if fname.endswith(".json")]
    log_files.sort()

    # display the hash of the last block
    for log_file in log_files:
        with open(f"{LOG_DIR}/{log_file}", 'r') as f:
            file_content = f.read()
        print(f"{log_file}: {hashlib.sha256(file_content.encode()).hexdigest()}")

Logs Comparation

The logs show that the gas consumption may wary from run to run. Usually the first divergence happen for the line line 1360 of the logs:

{"jsonrpc": "2.0", "id": 1, "result": {"cumulativeGasUsed": "0x9264d6", ...

{"jsonrpc": "2.0", "id": 1, "result": {"cumulativeGasUsed": "0x925ebe", ...

Further Details

The bug was discovered while my colleagues and I tried to emulate the block mining process. We collected a set of real transactions (smart contracts and externaly owned accounts) and tried to get the gas usege of the block when mined.

The payload.json file contains the payload data to call the hardhat RPC. The payloads description:

0. hardhat reset and fork call, done from the script

{
    "jsonrpc": "2.0",
    "method": "hardhat_reset",
    "params": [
        {
            "forking": {
                "jsonRpcUrl": JSON_RPC_TOKEN,
                "blockNumber": 20845407
            }
        }
    ],
    "id": 1
}

1. Turn off automine and get automine mode

[{"jsonrpc":"2.0","method":"evm_setAutomine","params":[false],"id":1},
{"jsonrpc":"2.0","method":"hardhat_getAutomine","params":[],"id":1}]

2. Set the block gas limit to the sum of all maxGasUsage

{"jsonrpc":"2.0","method":"evm_setBlockGasLimit","params":[205322942],"id":1}

3. Impersonate all accounts which are sent to the block

...
{"jsonrpc":"2.0","method":"hardhat_impersonateAccount","params":["0x894ff15a502f55943e488a6e035bff78c828b573"],"id":1},
{"jsonrpc":"2.0","method":"hardhat_impersonateAccount","params":["0xf89d7b9c864f589bbf53a82105107622b35eaa40"],"id":1},
...

4. This is quite non-standard scenario, the cause of the bag could be here. For our scenario we need to make sure that each transaction got to the pool as we are interested in the situation when, say, an account B does not have enough balance to send a transaction TxB, but if there is a transaction TxA which tops up the balance B making the block [TxA, TxB] a valid block. To bypass hardhat's automatic balance check (which prevents sending the tx TxB otherwise), for each transaction we do:
   • get the balance of the caller
   • set the balance of the caller to $2^{256} - 1$
   • submit the transaction to the pool
   • set the balance to the value we get from the first step

...
{"jsonrpc":"2.0","method":"eth_getBalance","params":["0x894fF15a502f55943E488a6E035bFF78c828B573","latest"],"id":1},
{"jsonrpc":"2.0","method":"hardhat_setBalance","params":["0x894fF15a502f55943E488a6E035bFF78c828B573","0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"],"id":1},
{"jsonrpc":"2.0","method":"eth_sendTransaction","params":[{"data":"0x095ea7b300000000000000000000000080a64c6d7f12c47b7c66c5b4e20e72bc1fcd5d9effffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff","from":"0x894fF15a502f55943E488a6E035bFF78c828B573","gas":57087,"nonce":683,"to":"0x65591F115286A284870e1677B1ad52DB4A762B54","value":0,"maxFeePerGas":12497504523,"maxPriorityFeePerGas":3000000000}],"id":1},
{"jsonrpc":"2.0","method":"hardhat_setBalance","params":["0x894fF15a502f55943E488a6E035bFF78c828B573","0x765f80d4ff56ca0"],"id":1},
...

5. mine the block

{"jsonrpc":"2.0","method":"evm_mine","params":[],"id":1}

6. for each transaction get the receipt and the new balance

...
{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params":["810962865bf8a0aef20a7653aa6be334860f9f83b82110a674888665d95c6886"],"id":1},
{"jsonrpc":"2.0","method":"eth_getBalance","params":["0xc5ed543899bd2cf38be5ae098c4987e5c900dc7c","0x13e1360"],"id":1},
...

Minimal reproduction steps

Reproducing the Bug

I managed to reproduce the bug in a container. You can find the PoC and instructions here. The reproduction instructions are:

1. git clone https://github.com/polka125/hardhat-bug-artifacts.git
2. put your remote api key to the file RPC_TOKEN
3. docker build -t hardhat_bug .
4. docker run -v ./host_logs:/app/logs -it hardhat_bug At this point you might get feeling that the docker stuck, but check host_logs/hardhat.log, it accumulates the hardhat output, i.e. the process is running
5. find ./host_logs -type f -print0 | sort -z | xargs -0 shasum -a 1

The last step will print hashes of the log files, the log files with different logs will be easy to locate visually as their hashes will diverge

Search terms

Gas usage, Fork mode

[polka125]

Hi, was it possible to reproduce the bug?

[fvictorio]

Hey @polka125, sorry for not responding this before, this issue fell through the cracks and we didn't prioritize it sooner. I'll take a look as soon as I can.

[fvictorio]

Hi @polka125. I haven't tried to reproduce this locally yet, but one thing I noticed is that you are not setting an initial timestamp. Since this changes between executions, it's possible that that's the reason for the difference in gas usage.

From what I can tell, you are executing the transactions of block 20845408. The timestamp of that block is 1727481635. So you can send evm_setNextBlockTimestamp with that value after hardhat_reset and before evm_mine to get a consistent timetsamp in the mined block.

Could you try that and tell me if the inconsistency persists?

[polka125]

Hi @fvictorio,
Oh, seems it was indeed the reason. I added the evm_setNextBlockTimestamp call, and now all blocks are identical.

Thank you!

[fvictorio]

Glad to hear that!