mds1 / convex-shutdown-simulation Goto Github PK

This sim relies on a lot of network requests, which are done over HTTPS. Foundry is adding websocket support, and if Hardhat and Ganache both support websockets as well we can add a separate table with websocket benchmarks

When foundry-rs/foundry#1003 is merged Forge will support WebSocket and IPC endpoints which should (in theory, at least) significantly speed up forking, since instead of closing and re-opening the RPC at every request, we would keep the connection alive.

As a sidenote, would it make sense to try and compare with a WebSocket endpoint as well? IPC might be a bit too much.

The transaction runs out of gas (hits block limit of 30M) during simulation with Ganache (I am using Alchemy as a node provider).
script output:

$  ./scripts/convex.ganache.ts
yarn run v1.22.15
Setting up Ganache...
  eth_chainId
  eth_chainId
  eth_call
  eth_accounts
  eth_sendTransaction

    Transaction: 0x22ba0b17256c047bf139d53b98a6cdbb8a1bee5ec10e1573883588c805e5c820
    Contract created: 0x324c828fbc98e722afd73ec91fbf4263eeccb0dc
    Gas usage: 64323
    Block number: 14121829
    Block time: Thu May 19 2022 18:26:28 GMT-0400 (Eastern Daylight Saving Time)

  eth_chainId
  eth_getTransactionReceipt
  eth_blockNumber
  eth_chainId
  eth_sendTransaction

    Transaction: 0xf9f9c5a0f511df85a01319364f0cfd157130c290eaf06fd89b263e43777a0245
    Contract created: 0xd62f3dfaf07eb471fbd4ab88bfe2c6c8a7dc8b3c
    Gas usage: 64323
    Block number: 14121830
    Block time: Thu May 19 2022 18:26:29 GMT-0400 (Eastern Daylight Saving Time)

  eth_chainId
  eth_getTransactionReceipt
  evm_addAccount
  personal_unlockAccount
setup-ganache: 3.101s

Simulating shutdown...
  eth_sendTransaction
  
    Transaction: 0x8617941537d4f6ddd9934752b93cbf7b335969f1c87956ccbfb110a018aeaaf9
    Gas usage: 30000000
    Block number: 14121831
    Block time: Thu May 19 2022 18:26:29 GMT-0400 (Eastern Daylight Saving Time)
    Runtime error: out of gas

I replicated the logic using Brownie with the same network parameters, but increased the block gas limit to 100M, and found it used around 60M gas:

Running 'scripts/sim.py::main'...
Executing system shutdown...
Transaction sent: 0x3007e01e625dd6e40d30890496ae5defc3206145de1636a7ee079f96b988d536
  Gas price: 0.0 gwei   Gas limit: 100000000   Nonce: 1
  Booster.shutdownSystem confirmed   Block: 14445963   Gas used: 60707716 (60.71%)

It would be great to know why my results differ from what's given in the readme.

Since logging was enabled in 2bb428a, you can now see in the run of the benchmark that Hardhat is running an eth_estimateGas call.

Discovered by @feuGeneA in #4, moving tracking of that issue here.

Ganache simulation fails with out of gas error, which @gnidan and @davidmurdoch are planning to look into. For reference, here's how much gas used is reported when simulating this transaction with various tools:

Gas usage at block 13,724,056:

• Foundry: 15,515,282 (using the fix in foundry-rs/foundry#524)
• Hardhat: 15,344,483
• Dapptools: 16,779,553 (I don't think this accounts for refunds)
• Tenderly: 15,537,677 (not in this repo, but will check it manually using their simulator)

For each tool, benchmark scripts should report

• full execution time, i.e. including tool startup time
• execution time of just the transaction simulation (not sure if this is possible for each tool)
• gas used
• some simple state verification, e.g. check logs or traces, or perhaps we just check this one manually for now?

Ideally the CI will run the benchmarks and save results somewhere. However, @alcuadrado has mentioned the performance of GH runners may fluctuate, though this may only occur between jobs and not within the same job. Before relying on CI to report the benchmarks, let's perform a series of benchmarks locally and within a single CI job and ensure the relative performance between tools is similar in the two

Some simulations query the node for this, some hardcode the address, and some just overwrite the owner with a local account. For consistency, let's just hardcode the owner address (the from address used) in each simulation for now