node-real / bsc-erigon Goto Github PK
View Code? Open in Web Editor NEWThis project forked from erigontech/erigon
Ethereum implementation on the efficiency frontier
License: GNU Lesser General Public License v3.0
This project forked from erigontech/erigon
Ethereum implementation on the efficiency frontier
License: GNU Lesser General Public License v3.0
ubuntu@ip-172-31-0-234:/data/erigon$ git remote -v
origin https://github.com/node-real/bsc-erigon.git (fetch)
origin https://github.com/node-real/bsc-erigon.git (push)
ubuntu@ip-172-31-0-234:/data/erigon$ git branch
* (HEAD detached at v1.0.2)
devel
ubuntu@ip-172-31-0-234:/data/erigon$ ./build/bin/erigon --version
erigon version 2.40.0-dev-3da15fcb
OS & Version: Linux (Ubuntu 22.04)
Commit hash:
Erigon Command (with flags/config):
./erigon/build/bin/erigon \
--datadir=/data/snapshot/erigon \
--nat any \
--http \
--http.addr=0.0.0.0 \
--http.port=8546 \
--http.vhosts=* \
--http.api=eth,trace,web3 \
--networkid=56 \
--chain=bsc \
--log.dir.path /data/log \
--metrics \
--metrics.addr=0.0.0.0 \
--metrics.port=6060
Concensus Layer: NA
Concensus Layer Command (with flags/config): NA
Chain/Network: bsc / 56
Syncing
stuck at bodies stage
launch the client, and sync stucks at a fixed block number.
INFO[04-13|09:39:15.654] HTTP endpoint opened url=[::]:8546 ws=false ws.compression=true grpc=false
INFO[04-13|09:39:15.657] [1/15 Snapshots] Fetching torrent files metadata
INFO[04-13|09:39:15.665] Started P2P networking version=68 self=enode://d6671bee3ebdf08e248546301fb680ecbe33f7451aadef054bc09c772e108f239dccc98c3325054cb3b2e0f91c5de75d7266808894d48a00faa3a720d1ec667f@127.0.0.1:30304 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.18.1
INFO[04-13|09:39:15.671] [snapshots] Blocks Stat blocks=23000k indices=23000k alloc=2.8GB sys=3.0GB
INFO[04-13|09:39:15.678] [2/15 Headers] Waiting for headers... from=27305933
INFO[04-13|09:39:15.698] Started P2P networking version=67 self=enode://d6671bee3ebdf08e248546301fb680ecbe33f7451aadef054bc09c772e108f239dccc98c3325054cb3b2e0f91c5de75d7266808894d48a00faa3a720d1ec667f@127.0.0.1:30303 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.18.1
INFO[04-13|09:39:28.713] [2/15 Headers] Processed highest inserted=27308655 age=42s
INFO[04-13|09:39:31.879] [5/15 Bodies] Processing bodies... from=22999999 to=27308655
INFO[04-13|09:39:51.881] [5/15 Bodies] No block bodies to write in this log period block number=22999999
INFO[04-13|09:40:11.880] [5/15 Bodies] No block bodies to write in this log period block number=22999999
INFO[04-13|09:40:15.636] [txpool] stat pending=1 baseFee=0 queued=14 alloc=3.3GB sys=3.5GB
INFO[04-13|09:40:31.881] [5/15 Bodies] No block bodies to write in this log period block number=22999999
Erigon version: ./erigon --version
v1.0.2
OS & Version: Windows/Linux/OSX
Ubuntu 20.04.5 LTS (GNU/Linux 5.15.0-1034-aws aarch64)
Erigon Command (with flags/config):
./build/bin/erigon --config="/root/config.toml"
chain="bsc"
datadir = "/erigon/bsc"
port="30303"
http="true"
ws="true"
"http.port"="8545"
"authrpc.port"="8551"
"torrent.port"="30303"
"private.api.addr"="127.0.0.1:9090"
"http.api"="web3,net,eth,debug,trace,txpool"
"http.corsdomain"="*"
"http.addr"="0.0.0.0"
"http.vhosts"="*"
"maxpeers"="100"
"p2p.protocol"="66"
"bodies.cache"="21474836480"
"batchSize"="8192M"
"rpc.gascap"="80000000"
"db.pagesize"="32k"
"bootnodes"=["enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311","enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311","enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311"]
"staticpeers"=["enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311","enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311","enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311"]
"staticpeers"=["enode://47de9d7808f339b55c5d958ba3a644c2423731de269fa926d8c78eb0b864e4c78734314dd1fc6439a99f1d4c0dab48d57f8a0bfa4b82ffcbf6547f880c41d079@52.202.229.96:30311", "enode://7287960657a7cd5a9e0e0cc6b4bb74110155979604d103929c5fcbfe6afc705c441d4b4cd2bdd0009f2ebb8185dab9fd78ef839af965a92c3ca5d45bd0303224@34.226.221.113:30311", "enode://627a1cb2c4712cce439026da0c2f599b97628c90c8ccc55526574a944b7455827544130b3003e79399cd79bd73a06a1d6bbd018fcf9ffc5297d3b731aa1b40ab@3.91.73.29:30311", "enode://16c7e98f78017dafeaa4129647d1ec66b32ee9be5ec753708820b7363091ceb310f575e7abd9603005e0e34d7b3316c1a4b6c8c42d7f074ed2eb4d073f800a03@3.85.216.212:30311", "enode://accbc0a5af0af03e1ec3b5e80544bdceea48011a6928cd82d2c1a9c38b65fd48ec970ba17bd8c0b0ec21a28faec9efe1d1ce55134784b9207146e2f62d8932ba@54.162.32.1:30311", "enode://c64c864572dae7ea25225a412c026ced0de66ae429b40c545be8f524a1aeb70b3441710dbfed19e3ba9ef08ce13b00a58daa7a7510924da8e6f4f412d8b45fd5@3.92.160.2:30311", "enode://5a838185d4b91eb42cbe3a60bb9f706484d8ec5041fa97b557d10e8ca10a459db0271e06e8b85cad57f1d2c7b05aa4319c0300b2936eefcb2302e10b253cf7d6@23.20.67.34:30311", "enode://3438d60bcb628ba33b0adf5e653751436fdc393a869fab136dec5ec6b2ed06d8ea30e4fec061f4f4a67bb01644897dbc3d14db44afc052eb69f102340aff70f9@18.215.252.114:30311", "enode://c307b4cddec0aea2188eafddedb0a076b9289402c63217b4c81eb7f34761c7cfaf6b075e93d7357169e226ff1bb4aa3bd71869b4c76cf261e2991005ddb4d4aa@3.81.81.182:30311", "enode://d69853daf3057cc191514afdf56df4769238fde4f261fab80c6e089480abb9916d61180e783d1cc9e5ae56d30ce6261d9954702dc73c41cd47e4b3961830b2dc@184.73.34.17:30311", "enode://ba88d1a8a5e849bec0eb7df9eabf059f8edeae9a9eb1dcf51b7768276d78b10d4ceecf0cde2ef191ced02f66346d96a36ca9da7d73542757d9677af8da3bad3f@54.198.97.197:30311", "enode://f7dc512940ca4a8f6858632abbdfc59cea6c4ed7a8da41ddfc4e4dac74e2664e74355fd7c688b285a22295e0053a800f759c9123ec741285a5bd602f89720cea@54.198.51.232:30311", "enode://bdbcb42ff17a52af7ac120f23ee86f525ffbd854ce76e54bad858cf741fcd524d0f810be399d437bea939682a919c5af5df31811c43ccc270d04485caf4ddaeb@52.206.226.126:30311", "enode://5fa49c3fc694fcba46199c4ac932a84a89435d545b04a3a68d47747fee41d417d8033c953f9c54ca943cb3d7eb82f936ab1f6ec93bb14ce466de4bcd50d410a5@44.201.87.43:30311", "enode://ace8e3b7e96290392a9c6238e005539dd1d1ca7c18aeedd979087789812e0656407e2c4baabcd83b50d583b583aa959ff42e5c95571d76b84995aad722a9a85c@44.198.55.182:30311", "enode://458c0e85ef43581557535e9fba2c8edef575737fd36476cb6b711461d74a9080fc38514e705311a788c0f034b2613839e0bd8ef82eafeb62d52cb5e845dd3e8f@3.250.75.234:30311", "enode://fe0bb07eae29e8cfaa5bb15b0db8c386a45b7da2c94e1dabd7ca58b6327eee0c27bdcea4f08db19ea07b9a1391e5496a28c675c6eee578154edae4fa44640c5d@54.228.2.74:30311", "enode://ca078d6849de674fe7fa0a7ca55057978566499d2c7401739d8ee6a8933a3ac3e3c29cfc6f8474e86dd576035ba0d92038115917f928d43c86e01eb761cac912@63.33.196.130:30311", "enode://a62ab1c9bbe97d8258a8944761933ad33891193e439feb84066e0fbe526a34aa7d3c5488f31f045c01890c111eff768cfff937e2edff18b824e47030a73add94@3.250.220.197:30311", "enode://1adabe43b638ec1fcd6559d4d4b765aae2826eae8a271418ff61c418e360da7e991c4b3099f1725fa9b157da3c8adf66117f918177367d59e679b99cb647003b@52.211.52.101:30311", "enode://322a42a08959aefd3423d17d8aeb802e0dbfb8bb0096aa712b6bf3036c91a80b0abc45c7a3d1320eda9a9c0337dd028967e4b84357080c258c8d0a3aaa02a821@34.245.12.138:30311", "enode://8ab18a0ad2872165710fdf907aa6c61ba163835d87475f6aa058c8e877cf2261ed93087e426d35fe2c10eee63d4e8dd6fb35cbb4b22a7346511a1024f87055a9@3.250.46.12:30311", "enode://a88322fa7db1958c4ce1c04e4980b7fdd23d2ea09ede072ffb487931dc62109cfad9defc2087568f625b4b5ac931c8f6f0baef37c988772efae2e12df3a30a70@52.19.216.114:30311", "enode://3aaaa0e0c7961ef3a9bf05f879f84308ca59651327cf94b64252f67448e582dcd6a6dbe996264367c8aa27fc302736db0283a3516c7406d48f268c5e317b9d49@34.250.1.192:30311", "enode://57824d2d9b5f39681bee265d56ec98a17fa4af343debdeba18596837f776f7c6370d8a33354e2b1750c41b221778e05c4189b93aca0d4cb1d45d32dc3b2d63f1@34.240.198.163:30311", "enode://67ec1f3df346e0aef401175119172e86a20e7ee1442cba4a2074519405cdae3708be3fdcb5e139094408b5d6f6c8e85f89ebb77d04833f7aa251c91344dbd4c9@3.249.178.199:30311", "enode://1afc9727301dcd8d2c5aef067031639ae3d3c7a23f8ba6c588a6a1b2c3cbcd738b4ccc53c07d08690ef591b99fd12f00a005f38d820354a91f418ab0939b9072@34.253.216.225:30311", "enode://3c13113538f3ca7d898d99f9656e0939451558758fd9c9475cff29f020187a56e8140bd24bd57164b07c3d325fc53e1ef622f793851d2648ed93d9d5a7ce975c@34.254.238.155:30311", "enode://5d54b9a5af87c3963cc619fe4ddd2ed7687e98363bfd1854f243b71a2225d33b9c9290e047d738e0c7795b4bc78073f0eb4d9f80f572764e970e23d02b3c2b1f@34.247.177.253:30311", "enode://1bb269476f62e99d17da561b1a6b0d0269b10afee029e1e9fdee9ac6a0e342ae562dfa8578d783109b80c0f100a19e03b057f37b2aff22d8a0aceb62020018fe@54.78.102.178:30311"]
Hi guys,
Thank you for your great work,
Since I started using node-real, it has never really synchronized in real time. I tried all possible solutions in the issue, but it still lags behind.
Can anyone actually sync in real time all the time? I'm wondering if there's something wrong with my configuration, or if it's just the program.
I've tried tinkering with the config, or using integration, but to no avail, it doesn't improve the lag a bit.
My machine has a 16-core 64G and has an 8T nvme, so performance should not be a problem.
Below is my running log, please help me, thanks.
Hi,
Decided to spin up a fresh bsc-erigon node today without using snapshots. Using v1.0.4 binary
At around 668.1GB chaindata size this error occured:
WARN[05-05|21:18:04.418] Index file has timestamp before segment file, will be recreated segfile=/root/dev/bsc-erigon/bsc-erigon-1.0.4/build/bin/node/snapshots/v1-021500-022000-headers.seg segtime=2023-05-05T21:18:04+0200 idxfile=v1-021500-022000-headers.idx idxtime=2023-05-05T20:49:58+0200
WARN[05-05|21:18:04.420] Index file has timestamp before segment file, will be recreated segfile=/root/dev/bsc-erigon/bsc-erigon-1.0.4/build/bin/node/snapshots/v1-022000-022500-headers.seg segtime=2023-05-05T21:10:16+0200 idxfile=v1-022000-022500-headers.idx idxtime=2023-05-05T19:30:22+0200
WARN[05-05|21:18:04.420] Index file has timestamp before segment file, will be recreated segfile=/root/dev/bsc-erigon/bsc-erigon-1.0.4/build/bin/node/snapshots/v1-022500-023000-headers.seg segtime=2023-05-05T21:11:16+0200 idxfile=v1-022500-023000-headers.idx idxtime=2023-05-05T12:15:07+0200
WARN[05-05|21:18:04.496] Index file has timestamp before segment file, will be recreated segfile=/root/dev/bsc-erigon/bsc-erigon-1.0.4/build/bin/node/snapshots/v1-022000-022500-bodies.seg segtime=2023-05-05T21:10:17+0200 idxfile=v1-022000-022500-bodies.idx idxtime=2023-05-05T20:49:57+0200
WARN[05-05|21:18:04.497] Index file has timestamp before segment file, will be recreated segfile=/root/dev/bsc-erigon/bsc-erigon-1.0.4/build/bin/node/snapshots/v1-022500-023000-bodies.seg segtime=2023-05-05T21:09:11+0200 idxfile=v1-022500-023000-bodies.idx idxtime=2023-05-05T20:49:57+0200
EROR[05-05|21:18:06.436] Staged Sync err="[1/15 Snapshots] BuildMissedIndices: HeadersIdx: at=v1-021500-022000-headers.seg, file: v1-021500-022000-headers.seg, runtime error: slice bounds out of range [:-1], [decompress.go:532 panic.go:884 panic.go:139 decompress.go:553 block_snapshots.go:1899 block_snapshots.go:1823 block_snapshots.go:869 block_snapshots.go:912 errgroup.go:75 asm_amd64.s:1594], [block_snapshots.go:1806 panic.go:884 decompress.go:532 panic.go:884 panic.go:139 decompress.go:553 block_snapshots.go:1899 block_snapshots.go:1823 block_snapshots.go:869 block_snapshots.go:912 errgroup.go:75 asm_amd64.s:1594]"
INFO[05-05|21:18:06.940] [1/15 Snapshots] Fetching torrent files metadata
INFO[05-05|21:18:26.945] [1/15 Snapshots] download progress="98.71% 659.5GB/668.1GB" download-time-left=0hrs:15m total-download-time=20s download=9.2MB/s upload=0B/s
INFO[05-05|21:18:26.946] [1/15 Snapshots] download peers=8 connections=10 files=170 alloc=4.4GB sys=6.6GB
INFO[05-05|21:18:46.944] [1/15 Snapshots] download progress="98.77% 659.9GB/668.1GB" download-time-left=0hrs:6m total-download-time=40s download=21.4MB/s upload=0B/s
INFO[05-05|21:18:46.944] [1/15 Snapshots] download peers=7 connections=8 files=170 alloc=2.7GB sys=6.6GB
INFO[05-05|21:18:54.998] [txpool] stat pending=1 baseFee=0 queued=184 alloc=3.0GB sys=6.6GB
INFO[05-05|21:19:06.945] [1/15 Snapshots] download progress="98.84% 660.4GB/668.1GB" download-time-left=0hrs:5m total-download-time=1m0s download=24.0MB/s upload=0B/s
INFO[05-05|21:19:06.945] [1/15 Snapshots] download peers=9 connections=10 files=170 alloc=3.3GB sys=6.6GB
It will spin like this in circles, deleting and re-downloading only to fail.
Unwinding execution by 10 blocks did not help
Node specs:
2TB SSD,
64gb ram,
14 cores
Netcup root dedicated server
For some common issue encountered, such as the follows:
No block header/body write
.Database is too large for current system
Please try to specify these flags in the command line for starting node:
--p2p.protocol=68
(bsc remove 66&67 after v1.4.x)--db.pagesize=16k
Also, for some cases that now work especially in reboot scenario, it'll help to unwind some blocks:
make integration
./build/bin/integration stage_exec --datadir ./data/ --chain bsc --unwind 10
Here is our command line to start on BSC Mainnet, you can use on your need.
Note: we have plenty SSD disk storage to support archive node, which is 14TB, so we set
--db.pagesize=16k
for fast DB performance. You may configure it according to your device setting.
./erigon --bodies.cache=214748364800 --batchSize=4096M --txpool.disable --metrics.addr=0.0.0.0 --log.console.verbosity=eror --log.dir.verbosity=dbug --http --ws --http.api=web3,net,eth,debug,trace,txpool,admin --http.addr=0.0.0.0 --db.pagesize=16k --datadir ${workspace}/data --private.api.addr=localhost:9090 --chain=bsc --metrics
Try not specify the body cache and batch size: --bodies.cache=214748364800 --batchSize=4096M
.
These 2 flags will bring heavy to RAM caches. Similar issue: #39
It won't affect sync. Two way to fix it.
Stop erigon and ./integration stage_headers --reset --chain=bsc --datadir=yourdata will fix.
If possible it would be great if the version numbers when queried could the could be changed to reflect that it is a fork, this would make it much easier for version tracking vs your repo.
It could still have reference to the base Erigon version but could also reflect the version number in the repo, something like:
bsc-erigon 1.0.2 upstream version 2.40.0-dev-3da15fcb maybe
Erigon version: ./erigon v1.0.3
OS & Version: Linux
Commit hash: 065538d5786ac0c83970c8213538924bc17e3c37
Erigon Command (with flags/config):
./erigon --p2p.protocol 66 --nodiscover --log.dir.path ./ --bodies.cache=214748364800 --batchSize=4096M --db.pagesize=16k --datadir ./ --private.api.addr=localhost:9099 --log.console.verbosity 3
Concensus Layer:
NA
Concensus Layer Command (with flags/config):
NA
Chain/Network:
NA
BSC Erigon node can start
[EROR] [04-26|04:08:07.617] Erigon startup err="mdbx_env_open: MDBX_TOO_LARGE: Database is too large for current system, e.g. could NOT be mapped into RAM, label: chaindata, trace: [kv_mdbx.go:266 node.go:323 node.go:326 backend.go:204 node.go:112 main.go:59 command.go:274 app.go:332 app.go:309 main.go:36 proc.go:250 asm_amd64.s:1594]"
Print the error, then exit.
NA
Erigon version: ./erigon v1.0.3
OS & Version: Linux
Commit hash: 065538d5786ac0c83970c8213538924bc17e3c37
Erigon Command (with flags/config):
./erigon --p2p.protocol 66 --nodiscover --log.dir.path ./ --db.size.limit=400GB --bodies.cache=214748364800 --batchSize=4096M --db.pagesize=16k --datadir ./ --private.api.addr=localhost:9099 --log.console.verbosity 3
Concensus Layer:
NA
Concensus Layer Command (with flags/config):
NA
Chain/Network:
NA
BSC Erigon node can run without crash.
It crashed after stuck in stage Header for a while.
Could have memory leak?
...
[DBUG] [04-26|04:31:15.231] [txpool] Commit written_kb=0 in=1.940005903s[INFO] [04-26|04:31:22.295] [p2p] GoodPeers eth66=1[INFO] [04-26|04:31:22.825] [txpool] stat pending=0 baseFee=0 queued=0 alloc=5.6GB sys=6.1GB[INFO] [04-26|04:31:25.453] [2/15 Headers] No block headers to write in this log period block number=199
fatal error: runtime: out of memory
runtime stack:
runtime.throw({0x28f4efc?, 0x2030?})
runtime/panic.go:1047 +0x5d fp=0x7fdd0d699ca8 sp=0x7fdd0d699c78 pc=0x47771d
runtime.sysMapOS(0xc173c00000, 0x400000?)
runtime/mem_linux.go:187 +0x11b fp=0x7fdd0d699cf0 sp=0x7fdd0d699ca8 pc=0x45617b
runtime.sysMap(0xc3ffffffff?, 0x7fdd3d048000?, 0x46b460?)
runtime/mem.go:142 +0x35 fp=0x7fdd0d699d20 sp=0x7fdd0d699cf0 pc=0x455b55
runtime.(*mheap).grow(0x4d6a580, 0xf?)
runtime/mheap.go:1468 +0x23d fp=0x7fdd0d699d90 sp=0x7fdd0d699d20 pc=0x46851d
runtime.(*mheap).allocSpan(0x4d6a580, 0xf, 0x0, 0x1)
runtime/mheap.go:1199 +0x1be fp=0x7fdd0d699e28 sp=0x7fdd0d699d90 pc=0x467c5e
runtime.(*mheap).alloc.func1()
runtime/mheap.go:918 +0x65 fp=0x7fdd0d699e70 sp=0x7fdd0d699e28 pc=0x4676e5
runtime.systemstack()
runtime/asm_amd64.s:492 +0x49 fp=0x7fdd0d699e78 sp=0x7fdd0d699e70 pc=0x4a9be9
Erigon version: ./erigon --version
erigon version 2.40.0-dev-3da15fcb
OS & Version: Linux
Erigon Command (with flags/config):
nohup ./build/bin/erigon --chain bsc --db.pagesize=16k --datadir=/server/bsc-erigon/data --http --ws --http.api=eth,debug,net,trace,web3,erigon --log.dir.path=/server/bsc-erigon/logs
Chain/Network: BSC
Start importing new blocks after downloading snapshot completes
Stuck at stat and goes nowhere, keep receiving error message:
keeps logging message like this: t=2023-03-31T03:19:08+0000 lvl=dbug msg="[p2p] Handshake failure" peer=c3276083eebc96a44003 err="network id does not match: theirs 1971, ours 56"
Tried adding --staticpeers flag but doesn’t work.
Just start erigon using above command, you will see the issue.
[backtrace]
Erigon version: ./erigon --version
OS & Version: Windows/Linux/OSX
Commit hash:
Erigon Command (with flags/config):
Concensus Layer:
Concensus Layer Command (with flags/config):
Chain/Network: chapel
I reconstruct a request of debug_traceCall from real tx 0x4569fec4b067d4ef214e025bca090a167a249e4a81f8068b03789ced082412d3
, but erigon return "execution Reverted", but our Meganode Archive can execute this transaction, I want to know the reason of Execution Reverted, and which one is correct?
curl --location 'localhost:8545' \ --header 'Content-Type: application/json' \ --data '{ "jsonrpc": "2.0", "id": 1, "method": "debug_traceCall", "params": [ { "from": "0xb4dd66d7c2c7e57f628210187192fb89d4b99dd4", "gas": "0x7fffffffffffffff", "gasPrice": "0x0", "callType": "call", "input": "0xf340fa01000000000000000000000000b4dd66d7c2c7e57f628210187192fb89d4b99dd4", "to": "0x0000000000000000000000000000000000001000", "value": "0x36ef88513e3f00" }, "0x1aeb816", { "tracer": "callTracer" } ] }'
{ "jsonrpc": "2.0", "id": 1, "result": { "from": "0x1284214b9b9c85549ab3d2b972df0deef66ac2c9", "gas": "0x2fa9cc8", "gasUsed": "0xc0fd", "to": "0x0000000000000000000000000000000000001000", "input": "0xf340fa010000000000000000000000001284214b9b9c85549ab3d2b972df0deef66ac2c9", "calls": [ { "from": "0x0000000000000000000000000000000000001000", "gas": "0x8fc", "gasUsed": "0x0", "to": "0x000000000000000000000000000000000000dead", "input": "0x", "value": "0x57e5a6e863980", "type": "CALL" } ], "value": "0x36ef88513e3f00", "type": "CALL" } }
{ "jsonrpc": "2.0", "id": 1, "result": { "from": "0x1284214b9b9c85549ab3d2b972df0deef66ac2c9", "gas": "0x23c2f3f8", "gasUsed": "0x5236", "to": "0x0000000000000000000000000000000000001000", "input": "0x", "error": "execution reverted", "value": "0x36ef88513e3f00", "type": "CALL" } }
[backtrace]
(I know running v1.0.3+ is not currently recommended for bsc mainnet; I was running a v1.0.4 node on bsc mainnet as a test, to find sync bugs like this one.)
[INFO] [05-08|17:13:53.950] [5/15 Bodies] Processed highest=28036306
[INFO] [05-08|17:13:53.950] [6/15 Senders] Started from=28035799 to=28036306
[INFO] [05-08|17:13:54.263] [7/15 Execution] Blocks execution from=28035799 to=28036306
[WARN] [05-08|17:13:54.321] [7/15 Execution] Execution failed block=28035800 hash=0xad1e9a9a2c1372b40a5a0e65209347edc01896a7830cfc87ad5b029266fdea58 err="mismatching validator list on epoch block"
[INFO] [05-08|17:13:54.321] UnwindTo block=28035799 bad_block_hash=0xad1e9a9a2c1372b40a5a0e65209347edc01896a7830cfc87ad5b029266fdea58
[INFO] [05-08|17:13:54.321] [7/15 Execution] Completed on block=28035799
[INFO] [05-08|17:13:54.388] [2/15 Headers] Waiting for headers... from=28035799
Visually, this is the same error as v1.0.3 had for chapel. Presumably has a different cause, given that the luban fork is inactive for bsc mainnet.
Epoch block https://bscscan.com/block/28035800 doesn't look any different than the previous epoch block https://bscscan.com/block/28035600, so I'm not sure what would trigger this for 28035800 but not 28035600.
It is likely that the not-inLuban consensus logic in v1.0.3+ is not functionally identical to the consensus logic in ≤v1.0.2.
Downgrading to v1.0.2 allows sync to proceed from 28035799 without any need for unwinding, so it seems that the validator state was not corrupted by some block between 28035600 and 28035799, but rather the failure is limited to the processing of 28035800.
BSC Erigon must follow the BSC upgrades, but currently only support full sync mode, validator mode is not supported since it has relatively lower priority, but cost more extra work and we do not have the resource to do it right now.
Besides the BSC hard forks, new features or strategies on how to support Erigon in long term can also be discussed here
Erigon version: erigon version 2.40.0-dev-3da15fcb
OS & Version: Linux
Erigon Command (with flags/config): nohup ./build/bin/erigon --chain chapel --db.pagesize=16k --datadir=/server/bsc-erigon/data --http --ws --http.api=eth,debug,net,trace,web3,erigon --log.dir.path=/server/bsc-erigon/logs
Chain/Network: Chapel
Get syned properly
Never started to sync
Simply run the above erigon command
Erigon version: release 1.0.2
OS & Version: ubuntu
Commit hash: 3da15fc
Erigon Command (with flags/config):
nohup ./bin/erigon \
--datadir=./data \
--chain=bsc \
--http.addr=0.0.0.0 \
--http.port=8545 \
--http.corsdomain=* \
--ws \
--http.vhosts=* \
--http.api=eth,net,trace,web3,erigon \
--torrent.download.rate 1024mb \
--prune hrtc \
--batchSize "1024M" \
--etl.bufferSize "1024M" \
--snap.stop \
--maxpeers 1000 \
--p2p.protocol=66 &
[INFO] [04-27|10:36:16.558] [2/15 Headers] Wrote block headers number=27711963 blk/second=2.650 alloc=4.3GB sys=9.6GB
[INFO] [04-27|10:36:18.280] [2/15 Headers] Processed highest inserted=27711998 age=2s
[INFO] [04-27|10:36:18.280] [2/15 Headers] DONE in=11m21.722558317s
14412 [INFO] [04-27|10:36:18.437] [5/15 Bodies] Processing bodies... from=27711681 to=27711998
[INFO] [04-27|10:36:35.081] [txpool] stat pending=2191 baseFee=55 queued=4828 alloc=4.4GB sys=9.6GB
[INFO] [04-27|10:36:38.438] [5/15 Bodies] Downloading block bodies block_num=27711681 delivery/sec=307.6KB wasted/sec=0B remaining=316 delivered=100 cache=35.0MB alloc=4.5GB sys=9.6GB
[INFO] [04-27|10:36:58.437] [5/15 Bodies] No block bodies to write in this log period block number=27711681
[INFO] [04-27|10:37:18.437] [5/15 Bodies] No block bodies to write in this log period block number=27711681
14417 [INFO] [04-27|10:37:33.597] [p2p] GoodPeers eth66=996
[INFO] [04-27|10:37:35.082] [txpool] stat pending=2193 baseFee=55 queued=4829 alloc=4.2GB sys=9.6GB
[INFO] [04-27|10:37:38.437] [5/15 Bodies] Downloading block bodies block_num=27711681 delivery/sec=173.5KB wasted/sec=0B remaining=316 delivered=162 cache=38.4MB alloc=4.2GB sys=9.6GB
[INFO] [04-27|10:37:58.438] [5/15 Bodies] No block bodies to write in this log period block number=27711681
[INFO] [04-27|10:37:58.438] [5/15 Bodies] DONE in=1m40.000854917s
14422 [INFO] [04-27|10:37:58.446] Commit cycle in=7.830907ms
[INFO] [04-27|10:37:58.446] Timings (slower than 50ms) Headers=11m21.722s BlockHashes=154ms Bodies=1m40s14424 [INFO] [04-27|10:37:58.446] Tables PlainState=229.7GB AccountChangeSet=2.2GB StorageChangeSet=5.9GB BlockTransaction=144.9GB TransactionLog=9.3GB FreeList=11.3MB Reclaimab leSpace=45.1GB
[INFO] [04-27|10:37:58.446] RPC Daemon notified of new headers from=27711681 to=27711998 hash=0x12f5542c45f33d42d30ce266bf7c8c93e604be71690c8f1a165817aa4e1e18f6 header sending=294.605µs log sending=2 60ns
14426 [INFO] [04-27|10:37:58.446] [2/15 Headers] Waiting for headers... from=27711998
[INFO] [04-27|10:37:59.331] [2/15 Headers] Processed highest inserted=27712014 age=55s
14428 [INFO] [04-27|10:37:59.338] [5/15 Bodies] Processing bodies... from=27711681 to=27712014
[INFO] [04-27|10:38:19.339] [5/15 Bodies] Downloading block bodies block_num=27711681 delivery/sec=520.0KB wasted/sec=0B remaining=332 delivered=175 cache=50.0MB alloc=4.5GB sys=9.6GB
[INFO] [04-27|10:38:35.081] [txpool] stat pending=2194 baseFee=55 queued=4830 alloc=4.6GB sys=9.6GB
[INFO] [04-27|10:38:39.339] [5/15 Bodies] No block bodies to write in this log period block number=27711681
[INFO] [04-27|10:38:59.339] [5/15 Bodies] No block bodies to write in this log period block number=27711681
[INFO] [04-27|10:39:19.339] [5/15 Bodies] No block bodies to write in this log period block number=27711681
14434 [INFO] [04-27|10:39:33.596] [p2p] GoodPeers eth66=991
[INFO] [04-27|10:39:35.082] [txpool] stat pending=2333 baseFee=55 queued=5024 alloc=4.4GB sys=9.6GB
[INFO] [04-27|10:39:39.339] [5/15 Bodies] No block bodies to write in this log period block number=27711681
[INFO] [04-27|10:39:39.339] [5/15 Bodies] DONE in=1m40.000868168s14438 [INFO] [04-27|10:39:39.340] Commit cycle in=851.89µs
[INFO] [04-27|10:39:39.340] Timings (slower than 50ms) Headers=884ms Bodies=1m40s14440 [INFO] [04-27|10:39:39.340] Tables PlainState=229.7GB AccountChangeSet=2.2GB StorageChangeSet=5.9GB BlockTransaction=144.9GB TransactionLog=9.3GB FreeList=11.3MB Reclaimab leSpace=45.1GB
[INFO] [04-27|10:39:39.341] RPC Daemon notified of new headers from=27711681 to=27712014 hash=0x8a979390fc5d5398735038c52252af33b77a12c0e275d61ce440618eb39ef32a header sending=330.602µs log sending=2 57ns
14442 [INFO] [04-27|10:39:39.341] [2/15 Headers] Waiting for headers... from=27712014
[INFO] [04-27|10:39:39.528] [2/15 Headers] Processed highest inserted=27712046 age=59s
14444 [INFO] [04-27|10:39:39.543] [5/15 Bodies] Processing bodies... from=27711681 to=27712046
[INFO] [04-27|10:39:59.544] [5/15 Bodies] Downloading block bodies block_num=27711681 delivery/sec=326.1KB wasted/sec=0B remaining=364 delivered=112 cache=58.0MB alloc=4.9GB sys=9.6GB
[INFO] [04-27|10:40:19.545] [5/15 Bodies] Downloading block bodies block_num=27711681 delivery/sec=277.1KB wasted/sec=0B remaining=364 delivered=204 cache=63.4MB alloc=5.2GB sys=9.6GB
[INFO] [04-27|10:40:35.082] [txpool] stat pending=2746 baseFee=55 queued=5560 alloc=5.5GB sys=9.6GB
[INFO] [04-27|10:40:39.544] [5/15 Bodies] No block bodies to write in this log period block number=2771168114449 [INFO] [04-27|10:40:59.544] [5/15 Bodies] No block bodies to write in this log period block number=27711681
[INFO] [04-27|10:41:19.544] [5/15 Bodies] No block bodies to write in this log period block number=27711681
[INFO] [04-27|10:41:19.544] [5/15 Bodies] DONE in=1m40.001151674s
14452 [INFO] [04-27|10:41:19.546] Commit cycle in=1.375297ms
Why should this feature exist?
What are the use-cases?
Do you have ideas regarding the implementation of this feature?
Are you willing to implement this feature?
Erigon version: 2.40.0-dev-2af7ec56
(builded from 1.0.2 tag)
OS & Version: Linux (Docker)
Commit hash: 3da15fc
Erigon Command (with flags/config):
command:
- "--datadir"
- "/srv/bsc/data/"
- "--ethash.dagdir"
- "/srv/bsc/data/"
- "--http"
- "--http.port"
- "8545"
- "--http.addr"
- "0.0.0.0"
- "--http.vhosts"
- "*"
- "--http.corsdomain"
- "bsc"
- "--http.api"
- "eth,net,web3,debug,admin,personal"
- "--authrpc.addr"
- "0.0.0.0"
- "--authrpc.port"
- "8551"
- "--authrpc.vhosts"
- "*"
- "--authrpc.jwtsecret"
- "/srv/bsc/data/jwt.hex"
- "--metrics"
- "--metrics.addr"
- "0.0.0.0"
- "--metrics.port"
- "6061"
- "--ws"
- "--maxpeers"
- "200"
- "--chain"
- "bsc"
- "--p2p.protocol"
- "66"
- "--bodies.cache"
- "214748364800"
- "--batchSize"
- "4096M"
- "--db.pagesize"
- "16k"
Concensus Layer: None (internal)
Concensus Layer Command (with flags/config): None
Chain/Network: bsc
Pass synchronization
Fail somewhere in MdbxKV at Senders stage.
[INFO] [04-30|07:13:26.479] [6/15 Senders] Flushed buffer file name=/srv/bsc/data/temp/erigon-sortable-buf-1490780490
[INFO] [04-30|07:13:50.951] [6/15 Senders] Recovery block_number=25226906 ch=0/10000
[INFO] [04-30|07:14:12.188] [txpool] stat pending=1 baseFee=0 queued=91 alloc=4.4GB sys=6.0GB
[INFO] [04-30|07:14:20.951] [6/15 Senders] Recovery block_number=25278648 ch=0/10000
[INFO] [04-30|07:14:31.752] [6/15 Senders] Flushed buffer file name=/srv/bsc/data/temp/erigon-sortable-buf-206732764
[INFO] [04-30|07:14:50.951] [6/15 Senders] Recovery block_number=25333132 ch=0/10000
[INFO] [04-30|07:15:11.399] [p2p] GoodPeers eth66=59
[INFO] [04-30|07:15:12.188] [txpool] stat pending=1 baseFee=0 queued=95 alloc=5.2GB sys=6.0GB
[INFO] [04-30|07:15:20.951] [6/15 Senders] Recovery block_number=25389859 ch=0/10000
[INFO] [04-30|07:15:36.363] [6/15 Senders] Flushed buffer file name=/srv/bsc/data/temp/erigon-sortable-buf-2527265756
[INFO] [04-30|07:15:50.951] [6/15 Senders] Recovery block_number=25446385 ch=0/10000
[INFO] [04-30|07:16:12.188] [txpool] stat pending=1 baseFee=0 queued=99 alloc=3.5GB sys=6.0GB
[EROR] [04-30|07:16:14.945] failed ReadTransactionByHash hash=0x6fec7acf1d28abbe42b1a51307f3036637c0fc5f7b13b2749f68ef19b9a93965 block=25487600 err="failed MdbxKV cursor.Next(): mdbx_cursor_get: MDBX_PAGE_NOTFOUND: Requested page not found"
[WARN] [04-30|07:16:14.945] [6/15 Senders] ReadCanonicalBodyWithTransactions can't find block num=25487600 hash=0x6fec7acf1d28abbe42b1a51307f3036637c0fc5f7b13b2749f68ef19b9a93965
[EROR] [04-30|07:16:15.307] Staged Sync err="[6/15 Senders] failed MdbxKV cursor.Next(): mdbx_cursor_get: MDBX_BAD_TXN: Transaction is not valid for requested operation, e.g. had errored and be must aborted, has a child, or is invalid"
[INFO] [04-30|07:16:16.062] [6/15 Senders] Started from=22999999 to=27793614
[INFO] [04-30|07:16:46.063] [6/15 Senders] Recovery block_number=23073075 ch=10000/10000
[INFO] [04-30|07:16:59.611] [6/15 Senders] Flushed buffer file name=/srv/bsc/data/temp/erigon-sortable-buf-2354223747
[INFO] [04-30|07:17:11.399] [p2p] GoodPeers eth66=60
[INFO] [04-30|07:17:12.189] [txpool] stat pending=1 baseFee=0 queued=105 alloc=4.7GB sys=8.4GB
[INFO] [04-30|07:17:16.062] [6/15 Senders] Recovery block_number=23145546 ch=9863/10000
[INFO] [04-30|07:17:41.839] [6/15 Senders] Flushed buffer file name=/srv/bsc/data/temp/erigon-sortable-buf-1626203942
[INFO] [04-30|07:17:46.062] [6/15 Senders] Recovery block_number=23223562 ch=9983/10000
I already checked out all my disks, raid array, memory and all what I can to do with hardware, platform seems alive. Dmesg and syslog doesn't reports about any issue. Also I tried to use snapshot - but actually it's broken and I did not decompress it.
Erigon version: 1.0.3 and devel
OS & Version: Linux
Erigon Command (with flags/config): /root/bsc-erigon/build/bin/erigon --rpc.batch.limit=10000 --datadir=/data/erigon/ --private.api.addr=0.0.0.0:9090 --chain bsc --http.addr=0.0.0.0 --http.vhosts=* --http.corsdomain=* --http.api=eth,erigon,web3,net,debug,trace,txpool --db.pagesize=16kb --ws --snap.stop
Chain/Network: BSC
Apr 21 09:27:55 rpc-bsc-erigon-prod-01 erigon[3255069]: [INFO] [04-21|09:27:55.776] [snapshots] Blocks Stat blocks=23000k indices=23000k alloc=3.0GB sys=3.2GB
Apr 21 09:27:55 rpc-bsc-erigon-prod-01 erigon[3255069]: [INFO] [04-21|09:27:55.776] [2/15 Headers] Waiting for headers... from=27538092
Apr 21 09:28:15 rpc-bsc-erigon-prod-01 erigon[3255069]: [INFO] [04-21|09:28:15.777] [2/15 Headers] Wrote block headers number=27538127 blk/second=1.750 alloc=2.5GB sys=4.6GB
Apr 21 09:28:35 rpc-bsc-erigon-prod-01 erigon[3255069]: [INFO] [04-21|09:28:35.776] [2/15 Headers] No block headers to write in this log period block number=27538127
Apr 21 09:28:55 rpc-bsc-erigon-prod-01 erigon[3255069]: [INFO] [04-21|09:28:55.723] [txpool] stat pending=0 baseFee=0 queued=0 alloc=3.2GB sys=4.9GB
Apr 21 09:28:55 rpc-bsc-erigon-prod-01 erigon[3255069]: [INFO] [04-21|09:28:55.777] [2/15 Headers] No block headers to write in this log period block number=27538127
Apr 21 09:29:15 rpc-bsc-erigon-prod-01 erigon[3255069]: [INFO] [04-21|09:29:15.779] [2/15 Headers] No block headers to write in this log period block number=27538127
Apr 21 09:29:35 rpc-bsc-erigon-prod-01 erigon[3255069]: [INFO] [04-21|09:29:35.776] [2/15 Headers] No block headers to write in this log period block number=27538127
I have 2 nodes. Both stopped to sync. Usually, it restart after some restart, but now it looks not restart to sync. Any idea ?
Erigon version: ./erigon --version
erigon version 2.40.0-dev-3da15fcb
OS & Version: Windows/Linux/OSX
Linux
Commit hash: 3da15fc
Erigon Command (with flags/config):
/usr/local/bin/erigon --snapshots=true \
--chain=bsc \
--datadir=/mnt/data/.bsc \
--http \
--http.addr=0.0.0.0 \
--http.port=8545 \
--http.api=eth,debug,net,trace,web3,erigon \
--log.dir.path=/home/ubuntu/bsc-mainnet/logs \
--log.dir.verbosity=info \
--torrent.upload.rate=16mb \
--torrent.download.rate=250mb \
--db.pagesize=16k
Concensus Layer:
Concensus Layer Command (with flags/config):
Chain/Network: 56
Can download snapshots
Always stuck at Waiting for torrents metadata: 1/138
t=2023-04-12T18:00:16+0700 lvl=info msg="[p2p] GoodPeers" eth66=31 eth68=3
t=2023-04-12T18:00:16+0700 lvl=info msg="[txpool] stat" pending=1 baseFee=0 queued=14 alloc=66.7MB sys=182.7MB
t=2023-04-12T18:00:16+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
t=2023-04-12T18:00:36+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
t=2023-04-12T18:00:56+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
t=2023-04-12T18:01:16+0700 lvl=info msg="[txpool] stat" pending=1 baseFee=0 queued=14 alloc=73.4MB sys=182.7MB
t=2023-04-12T18:01:16+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
t=2023-04-12T18:01:36+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
t=2023-04-12T18:01:56+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
t=2023-04-12T18:02:16+0700 lvl=info msg="[p2p] GoodPeers" eth66=31 eth68=3
t=2023-04-12T18:02:16+0700 lvl=info msg="[txpool] stat" pending=1 baseFee=0 queued=14 alloc=113.8MB sys=182.7MB
t=2023-04-12T18:02:16+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
t=2023-04-12T18:02:36+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
t=2023-04-12T18:02:56+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
t=2023-04-12T18:03:16+0700 lvl=info msg="[txpool] stat" pending=1 baseFee=0 queued=14 alloc=77.7MB sys=182.7MB
t=2023-04-12T18:03:16+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
t=2023-04-12T18:03:36+0700 lvl=info msg="[1/15 Snapshots] Waiting for torrents metadata: 1/138"
[backtrace]
I've been running a bsc-mainnet
node on v1.0.8, with a watchdog to restart it if it hits "no block bodies...", and it's been mostly stable; but every once in a while — seemingly at random (until I figured out the pattern) — it would come back up after a watchdog restart in a corrupted state, where execution would fail with mismatching validator list on epoch block
.
Fixing this when it happens, requires stopping the node and unwinding state_stages
by one block.
After seeing this a few times, I eventually noticed that this happens when, after a restart, the execution stage starts at a block height ending in {1,3,5,7,9}99
(i.e. one block before a consensus snapshot.)
I assume that the current logic is written to expect that some in-memory state in parlia will already be available from block ...98, to be used during execution of block ...99, to set up the snapshot for block ...00. Where if execution starts at block ...99, then that state won't be available, so the snapshot for block ...00 will end up calculated incorrectly.
hetzner AX101 - AMD 5950X 8TB NVME SSD
erigon version 2.40.0-dev-3da15fcb
Ubuntu 20.04 LTS
3da15fc ( release 1.0.2 )
/opt/erigon/bin/erigon --datadir=/opt/erigon/data-bsc --log.console.verbosity=4 --p2p.protocol=66 --db.pagesize=16k --chain bsc --batchSize=512M --nat extip:XX.XX.XX.XX --torrent.download.rate=50mb --torrent.upload.rate=1mb --maxpeers=400 --sentry.drop-useless-peers=true --txpool.locals=XXX --prune=hrtc --private.api.addr=127.0.0.1:9090 --http=false --trustedpeers=enode://f56dcbe59ddcf52e2abe5d5f5fded28bf823e7e2fb887cebbfe3c540ed0dfbbd778872e6b0c9c6243fcb79fdf3e1805ae98a7c389091e9cc55bfe6dedfce04b8@3.115.208.145:30311,enode://b4feb14a8247917f25a4603a0a3a58827e6e3954fa1fc0499f3e084476dcb2dc32e444e7c51cecbc1066d2c94062fc16aa80da1a008c94e576b67b84a3a111c5@13.112.103.141:30311,enode://7fed0d5ebfec2d68106cf91d4bbf2c794a22f12a11c18ef643818e8b8a5022f63abccfa50cb34fd30343530f67a70523525d94247b4f8d143dca7524d2ba8630@52.194.28.137:30311,enode://64e87612bf91e145e019a2cf877891973151ba0acfe822346d5f6876feb4b031f80b6ff2334d9fccc7522d4c27f4a0003cfc29e20db25f6eb89fc72f5d058d89@99.80.96.58:30311,enode://c67e08daecbef6e78832a1fb7eef09725ee6671aeb6dd63cd880b9a2075b945df64b4a6181bf8ed31d43bd7b77587c5380d61095e6d7989e3880656b2fb9448d@54.76.80.25:30311,enode://45ad31700cfd9bce487b912d4b10d8f657a6b4a12f46a71707a351f350a28ea9183fe38f8e4cbd4371972bd6f096072fa65bcf59c0ffb719a8ef83f403b4d656@52.18.62.124:30311,enode://935d02d00d9c5ecdc3bee7a56201eb68c9a9e2fc684ff1e606d56bebcb45722b3812df2c408bd74495140b92214e4bd28a00853641e581cbc3ebbe6ee6b2f794@50.17.94.194:30311,enode://8e68f76aef70929084fbcdc527357aa97cf0091ed80639ba8e5c35933e50034c22a0c6d30775ec9bfdeef21fc029bb895ac2221b97e1595d35110a5a27589089@54.157.26.59:30311,enode://3da255f8abdeaafe3e8acd8e861314782aec365216948f203b5da5fa5457e92ac7dd7519e2e95487d99d7158a1b47e276c6a23efefbe8da423dfe090578d3bec@3.218.173.35:30311,enode://47de9d7808f339b55c5d958ba3a644c2423731de269fa926d8c78eb0b864e4c78734314dd1fc6439a99f1d4c0dab48d57f8a0bfa4b82ffcbf6547f880c41d079@52.202.229.96:30311,enode://7287960657a7cd5a9e0e0cc6b4bb74110155979604d103929c5fcbfe6afc705c441d4b4cd2bdd0009f2ebb8185dab9fd78ef839af965a92c3ca5d45bd0303224@34.226.221.113:30311,enode://627a1cb2c4712cce439026da0c2f599b97628c90c8ccc55526574a944b7455827544130b3003e79399cd79bd73a06a1d6bbd018fcf9ffc5297d3b731aa1b40ab@3.91.73.29:30311,enode://16c7e98f78017dafeaa4129647d1ec66b32ee9be5ec753708820b7363091ceb310f575e7abd9603005e0e34d7b3316c1a4b6c8c42d7f074ed2eb4d073f800a03@3.85.216.212:30311,enode://accbc0a5af0af03e1ec3b5e80544bdceea48011a6928cd82d2c1a9c38b65fd48ec970ba17bd8c0b0ec21a28faec9efe1d1ce55134784b9207146e2f62d8932ba@54.162.32.1:30311,enode://c64c864572dae7ea25225a412c026ced0de66ae429b40c545be8f524a1aeb70b3441710dbfed19e3ba9ef08ce13b00a58daa7a7510924da8e6f4f412d8b45fd5@3.92.160.2:30311,enode://5a838185d4b91eb42cbe3a60bb9f706484d8ec5041fa97b557d10e8ca10a459db0271e06e8b85cad57f1d2c7b05aa4319c0300b2936eefcb2302e10b253cf7d6@23.20.67.34:30311,enode://3438d60bcb628ba33b0adf5e653751436fdc393a869fab136dec5ec6b2ed06d8ea30e4fec061f4f4a67bb01644897dbc3d14db44afc052eb69f102340aff70f9@18.215.252.114:30311,enode://c307b4cddec0aea2188eafddedb0a076b9289402c63217b4c81eb7f34761c7cfaf6b075e93d7357169e226ff1bb4aa3bd71869b4c76cf261e2991005ddb4d4aa@3.81.81.182:30311,enode://d69853daf3057cc191514afdf56df4769238fde4f261fab80c6e089480abb9916d61180e783d1cc9e5ae56d30ce6261d9954702dc73c41cd47e4b3961830b2dc@184.73.34.17:30311,enode://ba88d1a8a5e849bec0eb7df9eabf059f8edeae9a9eb1dcf51b7768276d78b10d4ceecf0cde2ef191ced02f66346d96a36ca9da7d73542757d9677af8da3bad3f@54.198.97.197:30311,enode://f7dc512940ca4a8f6858632abbdfc59cea6c4ed7a8da41ddfc4e4dac74e2664e74355fd7c688b285a22295e0053a800f759c9123ec741285a5bd602f89720cea@54.198.51.232:30311,enode://bdbcb42ff17a52af7ac120f23ee86f525ffbd854ce76e54bad858cf741fcd524d0f810be399d437bea939682a919c5af5df31811c43ccc270d04485caf4ddaeb@52.206.226.126:30311,enode://5fa49c3fc694fcba46199c4ac932a84a89435d545b04a3a68d47747fee41d417d8033c953f9c54ca943cb3d7eb82f936ab1f6ec93bb14ce466de4bcd50d410a5@44.201.87.43:30311,enode://ace8e3b7e96290392a9c6238e005539dd1d1ca7c18aeedd979087789812e0656407e2c4baabcd83b50d583b583aa959ff42e5c95571d76b84995aad722a9a85c@44.198.55.182:30311,enode://458c0e85ef43581557535e9fba2c8edef575737fd36476cb6b711461d74a9080fc38514e705311a788c0f034b2613839e0bd8ef82eafeb62d52cb5e845dd3e8f@3.250.75.234:30311,enode://fe0bb07eae29e8cfaa5bb15b0db8c386a45b7da2c94e1dabd7ca58b6327eee0c27bdcea4f08db19ea07b9a1391e5496a28c675c6eee578154edae4fa44640c5d@54.228.2.74:30311,enode://ca078d6849de674fe7fa0a7ca55057978566499d2c7401739d8ee6a8933a3ac3e3c29cfc6f8474e86dd576035ba0d92038115917f928d43c86e01eb761cac912@63.33.196.130:30311,enode://a62ab1c9bbe97d8258a8944761933ad33891193e439feb84066e0fbe526a34aa7d3c5488f31f045c01890c111eff768cfff937e2edff18b824e47030a73add94@3.250.220.197:30311,enode://1adabe43b638ec1fcd6559d4d4b765aae2826eae8a271418ff61c418e360da7e991c4b3099f1725fa9b157da3c8adf66117f918177367d59e679b99cb647003b@52.211.52.101:30311,enode://322a42a08959aefd3423d17d8aeb802e0dbfb8bb0096aa712b6bf3036c91a80b0abc45c7a3d1320eda9a9c0337dd028967e4b84357080c258c8d0a3aaa02a821@34.245.12.138:30311,enode://8ab18a0ad2872165710fdf907aa6c61ba163835d87475f6aa058c8e877cf2261ed93087e426d35fe2c10eee63d4e8dd6fb35cbb4b22a7346511a1024f87055a9@3.250.46.12:30311,enode://a88322fa7db1958c4ce1c04e4980b7fdd23d2ea09ede072ffb487931dc62109cfad9defc2087568f625b4b5ac931c8f6f0baef37c988772efae2e12df3a30a70@52.19.216.114:30311,enode://3aaaa0e0c7961ef3a9bf05f879f84308ca59651327cf94b64252f67448e582dcd6a6dbe996264367c8aa27fc302736db0283a3516c7406d48f268c5e317b9d49@34.250.1.192:30311,enode://57824d2d9b5f39681bee265d56ec98a17fa4af343debdeba18596837f776f7c6370d8a33354e2b1750c41b221778e05c4189b93aca0d4cb1d45d32dc3b2d63f1@34.240.198.163:30311,enode://67ec1f3df346e0aef401175119172e86a20e7ee1442cba4a2074519405cdae3708be3fdcb5e139094408b5d6f6c8e85f89ebb77d04833f7aa251c91344dbd4c9@3.249.178.199:30311,enode://1afc9727301dcd8d2c5aef067031639ae3d3c7a23f8ba6c588a6a1b2c3cbcd738b4ccc53c07d08690ef591b99fd12f00a005f38d820354a91f418ab0939b9072@34.253.216.225:30311,enode://3c13113538f3ca7d898d99f9656e0939451558758fd9c9475cff29f020187a56e8140bd24bd57164b07c3d325fc53e1ef622f793851d2648ed93d9d5a7ce975c@34.254.238.155:30311,enode://5d54b9a5af87c3963cc619fe4ddd2ed7687e98363bfd1854f243b71a2225d33b9c9290e047d738e0c7795b4bc78073f0eb4d9f80f572764e970e23d02b3c2b1f@34.247.177.253:30311,enode://1bb269476f62e99d17da561b1a6b0d0269b10afee029e1e9fdee9ac6a0e342ae562dfa8578d783109b80c0f100a19e03b057f37b2aff22d8a0aceb62020018fe@54.78.102.178:30311 --bootnodes=enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311,enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311,enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311
NA
NA
56
Transaction Propagation works with ~ +2 blocks
Transactions take minutes to hours after ~ 24 hours of uptime on the node.
Seems like the geth client has the same issue -> bnb-chain/bsc#1413
Could it be that parts of the network get isolated from the validators from time to time?
See bnb-chain/bsc#1419 -> about 318 of 390 connections match the erigon version on startup.
Erigon currently broadcast to a subset of connected nodes ( sqrt ), changing this to all available seems to help a bit, but the error still occurs, just later.
Start a node using the config posted above, try to broadcast after 24 hours or more.
[INFO] [04-19|09:34:38.885] [13/15 LogIndex] processing from=27479547 to=27480566
[INFO] [04-19|09:35:02.137] Got interrupt, shutting down...
[INFO] [04-19|09:35:02.137] Exiting Engine...
[INFO] [04-19|09:35:02.137] RPC server shutting down
[INFO] [04-19|09:35:02.138] RPC server shutting down
[WARN] [04-19|09:35:02.138] Failed to serve http endpoint err="http: Server closed"
[INFO] [04-19|09:35:02.138] Exiting...
[INFO] [04-19|09:35:02.138] RPC server shutting down
[WARN] [04-19|09:35:02.138] Failed to serve http endpoint err="http: Server closed"
systemd[1]: Stopping erigon...
erigon[3825005]: [INFO] [04-19|09:35:02.139] Engine HTTP endpoint close url=127.0.0.1:8551
erigon[3825005]: [INFO] [04-19|09:35:02.139] HTTP endpoint closed url=[::]:8545
systemd[1]: erigon.service: Succeeded.
systemd[1]: Stopped erigon.
systemd[1]: Started erigon.
erigon[3922505]: [WARN] [04-19|09:35:04.796] no log dir set, console logging only
Hi, Just curious to see if anyone would be willing to share their command line if they are running a stable archival node. My node is constantly trailing behind and I find myself frequently rebooting it to stay synced.
My current setup
Docker image ghcr.io/node-real/bsc-erigon:1.0.3
command:
- --chain=bsc
- --datadir=/root/.local/share/erigon
- --port=30303
- --private.api.addr=0.0.0.0:9090
- --torrent.port=42069
- --torrent.download.rate=1000mb
- --torrent.download.slots=6
- --p2p.protocol=66
- --downloader.verify
- --batchSize=512M
- --etl.bufferSize=512M
- --db.pagesize=16k
- --healthcheck
- --log.console.verbosity=info
Erigon version: ./erigon --version
ghcr.io/node-real/bsc-erigon:1.0.8
OS & Version: Windows/Linux/OSX
Ubuntu 22.04, but running docker
Commit hash:
ghcr.io/node-real/bsc-erigon:1.0.8
Erigon Command (with flags/config):
--chain bsc --snapshots=true --db.pagesize='16kb'
--datadir /home/erigon/.local/share/erigon
--http --http.addr 0.0.0.0 --http.port ${ERIGON_RPC_PORT:-8548} --http.compression --http.corsdomain '*' --http.vhosts '*' --http.api 'eth,net,web3,trace,debug,erigon,ots'
--ws --ws.compression
--rpc.accessList /erigon_config/rpc_rules.json
--torrent.download.rate 512mb --torrent.upload.rate 4mb
--port ${ERIGON_P2P_PORT:-30306} --authrpc.addr 0.0.0.0 --authrpc.port 8551 --private.api.addr 127.0.0.1:9090 --torrent.port 42069
--p2p.allowed-ports ${ERIGON_P2P_PORT:-30306},${ERIGON_P2P_PORT_B:-31306},${ERIGON_P2P_PORT_C:-32306},${ERIGON_P2P_PORT_D:-33306},${ERIGON_P2P_PORT_E:-34306}
--p2p.protocol=66
--maxpeers 400
--nat extip:${ERIGON_EXTIP}
--batchSize 4096M
--metrics --metrics.addr=0.0.0.0 --metrics.port=6060
--pprof --pprof.addr=0.0.0.0 --pprof.port=6061
--rpc.returndata.limit 5000000
--sentry.drop-useless-peers
--bootnodes enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311,enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311,enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311
--staticpeers enode://47de9d7808f339b55c5d958ba3a644c2423731de269fa926d8c78eb0b864e4c78734314dd1fc6439a99f1d4c0dab48d57f8a0bfa4b82ffcbf6547f880c41d079@52.202.229.96:30311,enode://7287960657a7cd5a9e0e0cc6b4bb74110155979604d103929c5fcbfe6afc705c441d4b4cd2bdd0009f2ebb8185dab9fd78ef839af965a92c3ca5d45bd0303224@34.226.221.113:30311,enode://627a1cb2c4712cce439026da0c2f599b97628c90c8ccc55526574a944b7455827544130b3003e79399cd79bd73a06a1d6bbd018fcf9ffc5297d3b731aa1b40ab@3.91.73.29:30311,enode://16c7e98f78017dafeaa4129647d1ec66b32ee9be5ec753708820b7363091ceb310f575e7abd9603005e0e34d7b3316c1a4b6c8c42d7f074ed2eb4d073f800a03@3.85.216.212:30311,enode://accbc0a5af0af03e1ec3b5e80544bdceea48011a6928cd82d2c1a9c38b65fd48ec970ba17bd8c0b0ec21a28faec9efe1d1ce55134784b9207146e2f62d8932ba@54.162.32.1:30311,enode://c64c864572dae7ea25225a412c026ced0de66ae429b40c545be8f524a1aeb70b3441710dbfed19e3ba9ef08ce13b00a58daa7a7510924da8e6f4f412d8b45fd5@3.92.160.2:30311,enode://5a838185d4b91eb42cbe3a60bb9f706484d8ec5041fa97b557d10e8ca10a459db0271e06e8b85cad57f1d2c7b05aa4319c0300b2936eefcb2302e10b253cf7d6@23.20.67.34:30311,enode://3438d60bcb628ba33b0adf5e653751436fdc393a869fab136dec5ec6b2ed06d8ea30e4fec061f4f4a67bb01644897dbc3d14db44afc052eb69f102340aff70f9@18.215.252.114:30311,enode://c307b4cddec0aea2188eafddedb0a076b9289402c63217b4c81eb7f34761c7cfaf6b075e93d7357169e226ff1bb4aa3bd71869b4c76cf261e2991005ddb4d4aa@3.81.81.182:30311,enode://d69853daf3057cc191514afdf56df4769238fde4f261fab80c6e089480abb9916d61180e783d1cc9e5ae56d30ce6261d9954702dc73c41cd47e4b3961830b2dc@184.73.34.17:30311,enode://ba88d1a8a5e849bec0eb7df9eabf059f8edeae9a9eb1dcf51b7768276d78b10d4ceecf0cde2ef191ced02f66346d96a36ca9da7d73542757d9677af8da3bad3f@54.198.97.197:30311,enode://f7dc512940ca4a8f6858632abbdfc59cea6c4ed7a8da41ddfc4e4dac74e2664e74355fd7c688b285a22295e0053a800f759c9123ec741285a5bd602f89720cea@54.198.51.232:30311,enode://bdbcb42ff17a52af7ac120f23ee86f525ffbd854ce76e54bad858cf741fcd524d0f810be399d437bea939682a919c5af5df31811c43ccc270d04485caf4ddaeb@52.206.226.126:30311,enode://5fa49c3fc694fcba46199c4ac932a84a89435d545b04a3a68d47747fee41d417d8033c953f9c54ca943cb3d7eb82f936ab1f6ec93bb14ce466de4bcd50d410a5@44.201.87.43:30311,enode://ace8e3b7e96290392a9c6238e005539dd1d1ca7c18aeedd979087789812e0656407e2c4baabcd83b50d583b583aa959ff42e5c95571d76b84995aad722a9a85c@44.198.55.182:30311,enode://458c0e85ef43581557535e9fba2c8edef575737fd36476cb6b711461d74a9080fc38514e705311a788c0f034b2613839e0bd8ef82eafeb62d52cb5e845dd3e8f@3.250.75.234:30311,enode://fe0bb07eae29e8cfaa5bb15b0db8c386a45b7da2c94e1dabd7ca58b6327eee0c27bdcea4f08db19ea07b9a1391e5496a28c675c6eee578154edae4fa44640c5d@54.228.2.74:30311,enode://ca078d6849de674fe7fa0a7ca55057978566499d2c7401739d8ee6a8933a3ac3e3c29cfc6f8474e86dd576035ba0d92038115917f928d43c86e01eb761cac912@63.33.196.130:30311,enode://a62ab1c9bbe97d8258a8944761933ad33891193e439feb84066e0fbe526a34aa7d3c5488f31f045c01890c111eff768cfff937e2edff18b824e47030a73add94@3.250.220.197:30311,enode://1adabe43b638ec1fcd6559d4d4b765aae2826eae8a271418ff61c418e360da7e991c4b3099f1725fa9b157da3c8adf66117f918177367d59e679b99cb647003b@52.211.52.101:30311,enode://322a42a08959aefd3423d17d8aeb802e0dbfb8bb0096aa712b6bf3036c91a80b0abc45c7a3d1320eda9a9c0337dd028967e4b84357080c258c8d0a3aaa02a821@34.245.12.138:30311,enode://8ab18a0ad2872165710fdf907aa6c61ba163835d87475f6aa058c8e877cf2261ed93087e426d35fe2c10eee63d4e8dd6fb35cbb4b22a7346511a1024f87055a9@3.250.46.12:30311,enode://a88322fa7db1958c4ce1c04e4980b7fdd23d2ea09ede072ffb487931dc62109cfad9defc2087568f625b4b5ac931c8f6f0baef37c988772efae2e12df3a30a70@52.19.216.114:30311,enode://3aaaa0e0c7961ef3a9bf05f879f84308ca59651327cf94b64252f67448e582dcd6a6dbe996264367c8aa27fc302736db0283a3516c7406d48f268c5e317b9d49@34.250.1.192:30311,enode://57824d2d9b5f39681bee265d56ec98a17fa4af343debdeba18596837f776f7c6370d8a33354e2b1750c41b221778e05c4189b93aca0d4cb1d45d32dc3b2d63f1@34.240.198.163:30311,enode://67ec1f3df346e0aef401175119172e86a20e7ee1442cba4a2074519405cdae3708be3fdcb5e139094408b5d6f6c8e85f89ebb77d04833f7aa251c91344dbd4c9@3.249.178.199:30311,enode://1afc9727301dcd8d2c5aef067031639ae3d3c7a23f8ba6c588a6a1b2c3cbcd738b4ccc53c07d08690ef591b99fd12f00a005f38d820354a91f418ab0939b9072@34.253.216.225:30311,enode://3c13113538f3ca7d898d99f9656e0939451558758fd9c9475cff29f020187a56e8140bd24bd57164b07c3d325fc53e1ef622f793851d2648ed93d9d5a7ce975c@34.254.238.155:30311,enode://5d54b9a5af87c3963cc619fe4ddd2ed7687e98363bfd1854f243b71a2225d33b9c9290e047d738e0c7795b4bc78073f0eb4d9f80f572764e970e23d02b3c2b1f@34.247.177.253:30311,enode://1bb269476f62e99d17da561b1a6b0d0269b10afee029e1e9fdee9ac6a0e342ae562dfa8578d783109b80c0f100a19e03b057f37b2aff22d8a0aceb62020018fe@54.78.102.178:30311
Concensus Layer:
Erigon
Concensus Layer Command (with flags/config):
n/a
Chain/Network: bsc
Normal Sync
Syncing works properly, then after the first iteration, new block bodies need to be downloaded as some time passed already (step 5). But instead of downloading all block bodies, only 100-200 are downloaded and step 5 is marked as DONE. All the other steps work properly.
Because of this the node falls behind as this is too slow.
Temporary solution is to restart the node. This is far from ideal, but the first iteration after restart downloads all block bodies again.
Happens every single time. Use the above docker image mentioned and my flags and use the latest official snapshot.
version: v2.40.0-dev-3da15fcb
logs:
[INFO] [04-12|11:23:33.893] [2/15 Headers] Waiting for headers... from=27281023
[INFO] [04-12|11:23:53.893] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:24:13.894] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:24:33.643] [txpool] stat pending=4722 baseFee=0 queued=22073 alloc=3.3GB sys=3.5GB
[INFO] [04-12|11:24:33.893] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:24:53.894] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:25:13.893] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:25:13.893] Req/resp stats req=0 reqMin=0 reqMax=0 skel=19 skelMin=27281022 skelMax=27318078 resp=952 respMin=27281022 respMax=27281022 dups=106
[INFO] [04-12|11:25:33.133] [p2p] GoodPeers eth66=19 eth67=1
[INFO] [04-12|11:25:33.660] [txpool] stat pending=4880 baseFee=0 queued=22110 alloc=3.3GB sys=3.9GB
[INFO] [04-12|11:25:33.893] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:25:33.893] Req/resp stats req=0 reqMin=0 reqMax=0 skel=19 skelMin=27281022 skelMax=27318078 resp=1014 respMin=27281022 respMax=27281022 dups=100
[INFO] [04-12|11:25:53.893] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:25:53.893] Req/resp stats req=0 reqMin=0 reqMax=0 skel=20 skelMin=27281022 skelMax=27318078 resp=974 respMin=27280978 respMax=27281022 dups=111
nohup ./build/bin/erigon --chain=bsc --datadir=/erigon_chain --http=true --http.addr=0.0.0.0 --http.port=9001 --http.api=eth,erigon,web3,net,debug,trace,txpool --ws --torrent.port=42069 --torrent.download.rate=32mb --authrpc.port=8551 --private.api.addr=127.0.0.1:9090 --db.pagesize=16kb --snapshots=true --rpc.batch.limit=10000 --prune=htc --prune.h.older=90000 --prune.t.older=90000 --prune.c.older=90000 &
curl -s -H "Content-Type:application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' 0.0.0.0:9001
{"jsonrpc":"2.0","id":1,"result":"0x0"}
Synchronized to the latest block:
[INFO] [05-22|02:57:18.958] [2/15 Headers] Wrote block headers number=28421656 blk/second=9.600 alloc=2.6GB sys=3.3GB
[INFO] [05-22|02:57:38.958] [2/15 Headers] No block headers to write in this log period block number=28421656
[INFO] [05-22|02:57:58.496] [p2p] GoodPeers eth66=31 eth68=1
[INFO] [05-22|02:57:58.905] [txpool] stat pending=1 baseFee=0 queued=14 alloc=2.7GB sys=3.3GB
[INFO] [05-22|02:57:58.958] [2/15 Headers] No block headers to write in this log period block number=28421656
[INFO] [05-22|02:58:18.958] [2/15 Headers] No block headers to write in this log period block number=28421656
[INFO] [05-22|02:58:38.958] [2/15 Headers] No block headers to write in this log period block number=28421656
[INFO] [05-22|02:58:58.905] [txpool] stat pending=1 baseFee=0 queued=14 alloc=2.5GB sys=3.3GB
[INFO] [05-22|02:58:58.958] [2/15 Headers] No block headers to write in this log period block number=28421656
[INFO] [05-22|02:59:18.958] [2/15 Headers] No block headers to write in this log period block number=28421656
Erigon version: 2.40.0-dev-3da15fcb
OS & Version: Linux
Commit hash:
Erigon Command (with flags/config): node/archive/bsc/bin/erigon --datadir=/node/archive/bsc/erigon/ --chain=bsc --db.read.concurrency 3000 --rpc.batch.concurrency 10000 --rpc.gascap=600000000 --rpc.evmtimeout=30s --rpc.batch.limit=100 --rpc.returndata.limit=104857600 --http=true --http.api eth,net,web3,debug,trace,txpool --http.addr 0.0.0.0 --http.corsdomain '*' --http.vhosts '*' --ws --authrpc.addr 127.0.0.1 --port=21040 --p2p.allowed-ports 21046,21048 --http.port 21041 --private.api.addr="" --authrpc.port=21043 --torrent.port=21044 --p2p.protocol=66 --maxpeers=150
Concensus Layer:
Concensus Layer Command (with flags/config):
Chain/Network: bsc
Hi, Since the BSC hard fork, memory usage has been very high, After half an hour and one hour of startup, the memory usage reaches over 60G.
I have already posted the specific situation in other posts.
the full archive node is too large, think about a more economy efficient support for it
Not determined yet, maybe we can only support for a limited block range
This should only be used in very rare cases e.g. if you are not 100% sure if something is a bug or asking a question that leads to improving the documentation. For general questions please use Erigon's discord.
blk only 2-4, forever can't catch up lastest block
The next upgrade on BSC testnet will be plato, which coming soon.
You may refer: https://forum.bnbchain.org/t/bnb-chain-upgrades-testnet/934#platoupcoming-3
BSC Erigon need port some hard fork changes to support it.
According to the BSC Plato testnet release: bnb-chain/bsc#1596, need to port several PRs to Erigon.
Here is the list:
v1.0.6 failed to sync the BSC testnet upgrade: Plato, refer: https://github.com/node-real/bsc-erigon/issues
It is caused by a miss handle of the receipt of FastFinality reward distribution.
https://testnet.bscscan.com/txs?block=29861000
// Plato on 29861024
https://testnet.bscscan.com/txs?block=29861200
after Plato, there is a new system transaction: distributeFastFinalityReward, it was not handled correctly.
a.get the latest release: v1.0.7(not yet)
b.reset your node to make sure it clear up the dirty state
//== check you latest stages:
./build/bin/integration print_stages --chain=chapel --datadir=<datadir>
//== reset hash & trie
./build/bin/integration stage_hash_state --datadir=<datadir> --reset --chain=chapel
./build/bin/integration stage_trie --datadir=<datadir> --reset --chain=chapel
//== unwind to the block before Plato(29861024)
./build/bin/integration state_stages --unwind=20 --datadir=<datadir> --chain=chapel
//== start with the latest release
<just use your original command line>
Erigon version: 1.0.8
OS & Version: Ubuntu 22.04
Erigon Command (with flags/config):
/home/blockchain/bsc-testnet/erigon \
--chain=chapel \
--snapshots=true \
--datadir=/home/blockchain/bsc-testnet/chaindata \
--db.pagesize=4kb \
--port=30304 \
--authrpc.port=8551 \
--maxpeers=1000 \
--private.api.addr=127.0.0.1:9090 \
--bootnodes="enode://69a90b35164ef862185d9f4d2c5eff79b92acd1360574c0edf36044055dc766d87285a820233ae5700e11c9ba06ce1cf23c1c68a4556121109776ce2a3990bba@52.199.214.252:30311","enode://330d768f6de90e7825f0ea6fe59611ce9d50712e73547306846a9304663f9912bf1611037f7f90f21606242ded7fb476c7285cb7cd792836b8c0c5ef0365855c@18.181.52.189:30311","enode://df1e8eb59e42cad3c4551b2a53e31a7e55a2fdde1287babd1e94b0836550b489ba16c40932e4dacb16cba346bd442c432265a299c4aca63ee7bb0f832b9f45eb@52.51.80.128:30311","enode://0bd566a7fd136ecd19414a601bfdc530d5de161e3014033951dd603e72b1a8959eb5b70b06c87a5a75cbf45e4055c387d2a842bd6b1bd8b5041b3a61bab615cf@34.242.33.165:30311","enode://604ed87d813c2b884ff1dc3095afeab18331f3cc361e8fb604159e844905dfa6e4c627306231d48f46a2edceffcd069264a89d99cdbf861a04e8d3d8d7282e8a@3.209.122.123:30311","enode://4d358eca87c44230a49ceaca123c89c7e77232aeae745c3a4917e607b909c4a14034b3a742960a378c3f250e0e67391276e80c7beee7770071e13f33a5b0606a@52.72.123.113:30311" \
--torrent.port=42069 \
--metrics --metrics.port=8676 --metrics.addr=0.0.0.0
Chain/Network: Chapel Testnet
Node syncs from 0 to the latest block smoothly.
After downloading headers & bodies and recovering senders, execution fails at block 90:
Just start testnet node with empty chaindata dir.
[INFO] [05-19|17:17:43.137] [5/15 Bodies] Processed highest=29939319
[INFO] [05-19|17:17:43.137] [5/15 Bodies] DONE in=1h31m44.252406649s
[INFO] [05-19|17:17:43.139] [6/15 Senders] Started from=89 to=29939319
...
[INFO] [05-19|17:50:35.407] [6/15 Senders] ETL [2/2] Loading into=TxSender block=25821735
[INFO] [05-19|17:50:39.886] [txpool] stat pending=0 baseFee=0 queued=31 alloc=5.5GB sys=14.4GB
[INFO] [05-19|17:50:41.011] [6/15 Senders] DONE in=32m57.873295154s
[INFO] [05-19|17:50:41.011] [7/15 Execution] Blocks execution from=89 to=29939319
[WARN] [05-19|17:50:41.027] [7/15 Execution] Execution failed block=90 hash=0xc137446336ab18878949947a44d234afeb37ebff7bf5b70c64eb221652b3a86e err="expected system tx (hash 0xc27cd46b32b596c89b7a625ec6aea651992bb330ee34d355c558aaaf5fee731d, nonce 2, to 0x0000000000000000000000000000000000001002, value 0x8a7c0b96a5c0, gas 9223372036854775807, gasPrice 0x0, data ), actual tx (hash 0xff3c7d2fd274b74dce5b23650bc42d5eded31b2729d366013b13ae45323f6fbc, nonce 2, to 0x0000000000000000000000000000000000001002, value 0x205375a50b480, gas 9223372036854775807, gasPrice 0x0, data )"
[INFO] [05-19|17:50:41.027] UnwindTo block=89 bad_block_hash=0xc137446336ab18878949947a44d234afeb37ebff7bf5b70c64eb221652b3a86e
[INFO] [05-19|17:50:41.027] [7/15 Execution] Completed on block=89
[INFO] [05-19|17:51:01.028] [5/15 Bodies] Unwinding transactions... current block=2041233
...
[INFO] [05-19|18:16:20.455] [2/15 Headers] Unwind done in=4m25.871385824s
[INFO] [05-19|18:16:20.457] Timings (slower than 50ms) Headers=4m17.974s CumulativeIndex=62ms BlockHashes=910ms Bodies=1h31m44.252s Senders=32m57.873s Unwind Bodies=21m13.555s Unwind Headers=4m25.871s
[INFO] [05-19|18:16:35.065] RPC Daemon notified of new headers from=0 to=29939319 hash=0x0000000000000000000000000000000000000000000000000000000000000000 header sending=14.606902245s log sending=241ns
[INFO] [05-19|18:16:35.066] [2/15 Headers] Waiting for headers... from=89
[INFO] [05-19|18:16:39.885] [txpool] stat pending=0 baseFee=0 queued=31 alloc=11.5GB sys=18.5GB
[INFO] [05-19|18:16:55.066] [2/15 Headers] No block headers to write in this log period block number=89
[INFO] [05-19|18:17:09.223] New txs subscriber joined
[INFO] [05-19|18:17:09.223] new subscription to newHeaders established
[INFO] [05-19|18:17:15.066] [2/15 Headers] No block headers to write in this log period block number=89
[INFO] [05-19|18:17:35.066] [2/15 Headers] No block headers to write in this log period block number=89
Erigon version: v1.0.5
OS & Version: Linux/amd64 (Ubuntu 22.04)
Node reaches upgrade block, upgrade fails; node is restarted; consensus-engine in upgraded regime seems to fail to process existing snapshot state data, then fails again:
[INFO] [05-16|22:28:40.143] [2/15 Headers] Waiting for headers... from=29861022
[INFO] [05-16|22:28:43.076] [2/15 Headers] Processed highest inserted=29861023 age=0
[INFO] [05-16|22:28:43.094] [7/15 Execution] Completed on block=29861023
[INFO] [05-16|22:28:43.102] Commit cycle in=2.170581ms
[INFO] [05-16|22:28:43.102] Timings (slower than 50ms) Headers=2.933s
[INFO] [05-16|22:28:43.102] Tables PlainState=32.8GB AccountChangeSet=16.4GB StorageChangeSet=69.0GB BlockTransaction=132.1GB TransactionLog=87.7GB Fr>
[INFO] [05-16|22:28:43.102] RPC Daemon notified of new headers from=29861022 to=29861023 hash=0x566c57e3b41a07e504bc3305516eb50797814a766e6b9ddb0c1ecffc2b50e12a header sending=10>
[INFO] [05-16|22:28:43.103] [2/15 Headers] Waiting for headers... from=29861023
[EROR] [05-16|22:28:46.112] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861024 blockHash=0x5d78b2201c37e72a5d3d33a4182aaa6cb0bf4132ad5aa01afe179759a>
[INFO] [05-16|22:28:46.113] [2/15 Headers] Processed highest inserted=29861024 age=0
[INFO] [05-16|22:28:46.114] [7/15 Execution] Completed on block=29861024
[EROR] [05-16|22:28:46.114] [9/15 IntermediateHashes] Wrong trie root of block 29861024: 74c3fec75ed11d77d2d3f5c9285f1262cf262217204899218bdcc8632d50c2a2, expected (from header): 7d0c4>
[WARN] [05-16|22:28:46.114] Unwinding due to incorrect root hash to=29861023
[INFO] [05-16|22:28:46.114] UnwindTo block=29861023 bad_block_hash=0x5d78b2201c37e72a5d3d33a4182aaa6cb0bf4132ad5aa01afe179759a8a4d3ff
[INFO] [05-16|22:28:46.114] [8/15 HashState] Unwinding started from=29861024 to=29861023 storage=false codes=true
[INFO] [05-16|22:28:46.114] [8/15 HashState] Unwinding started from=29861024 to=29861023 storage=false codes=false
[INFO] [05-16|22:28:46.114] [8/15 HashState] Unwinding started from=29861024 to=29861023 storage=true codes=false
[INFO] [05-16|22:28:46.115] [7/15 Execution] Unwind Execution from=29861024 to=29861023
[INFO] [05-16|22:28:46.115] [2/15 Headers] Waiting for headers... from=29861023
[WARN] [05-16|22:28:46.129] [downloader] Rejected header marked as bad hash=0x5d78b2201c37e72a5d3d33a4182aaa6cb0bf4132ad5aa01afe179759a8a4d3ff height=29861024
[WARN] [05-16|22:28:46.171] [downloader] Rejected header marked as bad hash=0x5d78b2201c37e72a5d3d33a4182aaa6cb0bf4132ad5aa01afe179759a8a4d3ff height=29861024
[WARN] [05-16|22:28:46.174] [downloader] Rejected header marked as bad hash=0x5d78b2201c37e72a5d3d33a4182aaa6cb0bf4132ad5aa01afe179759a8a4d3ff height=29861024
...
Stopping Erigon blockchain node...
[INFO] [05-16|22:31:47.323] Got interrupt, shutting down...
[INFO] [05-16|22:31:47.323] Exiting Engine...
[INFO] [05-16|22:31:47.323] Exiting...
[INFO] [05-16|22:31:47.323] RPC server shutting down
[WARN] [05-16|22:31:47.323] Failed to serve http endpoint err="http: Server closed"
[INFO] [05-16|22:31:47.323] HTTP endpoint closed url=127.0.0.1:9545
[INFO] [05-16|22:31:47.323] RPC server shutting down
[INFO] [05-16|22:31:47.323] RPC server shutting down
[WARN] [05-16|22:31:47.323] Failed to serve http endpoint err="http: Server closed"
[INFO] [05-16|22:31:47.323] Engine HTTP endpoint close url=127.0.0.1:9551
[email protected]: Deactivated successfully.
Stopped Erigon blockchain node.
[email protected]: Consumed 30min 23.843s CPU time.
Started Erigon blockchain node.
[INFO] [05-16|22:31:57.980] logging to file system log dir=/scratch/node/chapel/logs file prefix=erigon log level=crit json=false
[INFO] [05-16|22:31:57.985] Build info git_branch= git_tag= git_commit=
[INFO] [05-16|22:31:57.985] Starting Erigon on Chapel testnet...
[INFO] [05-16|22:31:57.986] Maximum peer count ETH=200 total=200
[INFO] [05-16|22:31:57.986] starting HTTP APIs APIs=admin,debug,eth,net,web3,erigon
[INFO] [05-16|22:31:57.986] torrent verbosity level=WRN
[INFO] [05-16|22:31:57.986] [torrent] Public IP ip=15.235.54.154
[INFO] [05-16|22:31:57.987] Set global gas cap cap=50000000
[INFO] [05-16|22:31:57.991] [Downloader] Runnning with ipv6-enabled=true ipv4-enabled=true download.rate=64mb upload.rate=4mb
[INFO] [05-16|22:31:57.991] Opening Database label=chaindata path=/scratch/node/chapel/chaindata
[INFO] [05-16|22:31:57.991] [db] params: growStep=2GB, mapsSize=7TB, shrinkThreshold=-1, pageSize=16KB, label=chaindata, WriteMap=false, Durable=false, NoReadahead=true,
[INFO] [05-16|22:31:57.992] Initialised chain configuration config="{ChainID: 97 Ramanujan: 1010000, Niels: 1014369, MirrorSync: 5582500, Bruno: 13837000, Euler: 19203503, Gib>
[WARN] [05-16|22:31:57.992] Incorrect snapshot enablement got=true change_to=false
[INFO] [05-16|22:31:57.992] Effective prune_flags="--prune.c.older=90000" snapshot_flags= history.v3=false
[INFO] [05-16|22:31:57.994] Initialising Ethereum protocol network=97
[INFO] [05-16|22:31:58.408] Starting private RPC server on=127.0.0.1:10097
[INFO] [05-16|22:31:58.408] new subscription to logs established
[INFO] [05-16|22:31:58.408] rpc filters: subscribing to Erigon events
[INFO] [05-16|22:31:58.409] new subscription to newHeaders established
[INFO] [05-16|22:31:58.409] New txs subscriber joined
[INFO] [05-16|22:31:58.409] Reading JWT secret path=/scratch/node/chapel/jwt.hex
[INFO] [05-16|22:31:58.410] HTTP endpoint opened for Engine API url=127.0.0.1:9551 ws=true ws.compression=true
[INFO] [05-16|22:31:58.410] HTTP endpoint opened url=127.0.0.1:9545 ws=true ws.compression=true grpc=false
[INFO] [05-16|22:31:58.414] Started P2P networking version=67 self=enode://412e32aca65b49a8f35c29214fbb5b0b4a7a44554d0e10fc7c71ffd32caa00ca6e46cb2f0d0f23a6b3f1f8abf7c>
[INFO] [05-16|22:31:58.414] Started P2P networking version=66 self=enode://412e32aca65b49a8f35c29214fbb5b0b4a7a44554d0e10fc7c71ffd32caa00ca6e46cb2f0d0f23a6b3f1f8abf7c>
[INFO] [05-16|22:31:58.417] Started P2P networking version=68 self=enode://412e32aca65b49a8f35c29214fbb5b0b4a7a44554d0e10fc7c71ffd32caa00ca6e46cb2f0d0f23a6b3f1f8abf7c>
[INFO] [05-16|22:31:58.419] [txpool] Started
[INFO] [05-16|22:31:58.419] [2/15 Headers] Waiting for headers... from=29861023
[INFO] [05-16|22:32:01.333] New txs subscriber joined
[INFO] [05-16|22:32:01.333] new subscription to newHeaders established
[INFO] [05-16|22:32:58.409] [txpool] stat pending=13 baseFee=0 queued=12 alloc=91.9MB sys=208.4MB
[INFO] [05-16|22:33:57.994] [p2p] GoodPeers eth67=2 eth66=7
[INFO] [05-16|22:33:58.409] [txpool] stat pending=45 baseFee=0 queued=34 alloc=83.7MB sys=212.9MB
[INFO] [05-16|22:34:12.726] [2/15 Headers] Inserting headers progress=29861023 queue=1
[EROR] [05-16|22:34:12.728] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861025 blockHash=0xcff232a0021469422520093a26ad86a95e8c58fc533c9d97dbceb5810>
[EROR] [05-16|22:34:12.731] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861026 blockHash=0x82506ae10091cdaf01811487b8c2a3ae5be81f7c99120009baa1e0b53>
[EROR] [05-16|22:34:12.731] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861026 blockHash=0x82506ae10091cdaf01811487b8c2a3ae5be81f7c99120009baa1e0b53>
[EROR] [05-16|22:34:12.733] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861027 blockHash=0x7f5f5a7760aff9ec6dae628f0893c17454ef64fb89b5b058822ad2a86>
[EROR] [05-16|22:34:12.733] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861027 blockHash=0x7f5f5a7760aff9ec6dae628f0893c17454ef64fb89b5b058822ad2a86>
[EROR] [05-16|22:34:12.735] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861028 blockHash=0xfba701379efc76254003876ae4aad7e37c707aa06e23cae64753d5514>
[EROR] [05-16|22:34:12.735] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861028 blockHash=0xfba701379efc76254003876ae4aad7e37c707aa06e23cae64753d5514>
[EROR] [05-16|22:34:12.737] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861029 blockHash=0xc04294d88837a0a21a6e829ae28bc512707fcc8cd3210dfcb9f5a4d7e>
...
[EROR] [05-16|22:34:12.863] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861092 blockHash=0x8c6b3e699d16ee084a53ce1a8e0923e6802d49a9709632db275fa7f18>
[EROR] [05-16|22:34:12.863] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861092 blockHash=0x8c6b3e699d16ee084a53ce1a8e0923e6802d49a9709632db275fa7f18>
[EROR] [05-16|22:34:12.866] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861093 blockHash=0x148ecf0c8955137cf4d45ca49b8779123201052cfd808b08d1b3df158>
[EROR] [05-16|22:34:12.866] Unexpected error when getting snapshot error="unknown ancestor" blockNumber=29861093 blockHash=0x148ecf0c8955137cf4d45ca49b8779123201052cfd808b08d1b3df158>
[INFO] [05-16|22:34:12.867] [2/15 Headers] Processed highest inserted=29861093 age=1m59s
[INFO] [05-16|22:34:12.867] [2/15 Headers] DONE in=2m14.448336512s
[INFO] [05-16|22:34:12.889] [5/15 Bodies] Processing bodies... from=29861023 to=29861093
[INFO] [05-16|22:34:12.937] [5/15 Bodies] Processed highest=29861093
[INFO] [05-16|22:34:12.937] [6/15 Senders] Started from=29861023 to=29861093
[INFO] [05-16|22:34:12.940] [7/15 Execution] Blocks execution from=29861023 to=29861093
[WARN] [05-16|22:34:12.941] [7/15 Execution] Execution failed block=29861025 hash=0xcff232a0021469422520093a26ad86a95e8c58fc533c9d97dbceb581026f2628 err="mismatched receipt head>
[INFO] [05-16|22:34:12.941] UnwindTo block=29861024 bad_block_hash=0xcff232a0021469422520093a26ad86a95e8c58fc533c9d97dbceb581026f2628
[INFO] [05-16|22:34:12.941] [7/15 Execution] Completed on block=29861024
[INFO] [05-16|22:34:12.942] Timings (slower than 50ms) Headers=2m14.448s
[INFO] [05-16|22:34:12.942] RPC Daemon notified of new headers from=29861023 to=29861093 hash=0x0000000000000000000000000000000000000000000000000000000000000000 header sending=42>
[INFO] [05-16|22:34:12.942] [2/15 Headers] Waiting for headers... from=29861024
[WARN] [05-16|22:34:12.959] [downloader] Rejected header marked as bad hash=0xcff232a0021469422520093a26ad86a95e8c58fc533c9d97dbceb581026f2628 height=29861025
[WARN] [05-16|22:34:12.971] [downloader] Rejected header marked as bad hash=0xcff232a0021469422520093a26ad86a95e8c58fc533c9d97dbceb581026f2628 height=29861025
[WARN] [05-16|22:34:12.973] [downloader] Rejected header marked as bad hash=0xcff232a0021469422520093a26ad86a95e8c58fc533c9d97dbceb581026f2628 height=29861025
[WARN] [05-16|22:34:12.975] [downloader] Rejected header marked as bad hash=0xcff232a0021469422520093a26ad86a95e8c58fc533c9d97dbceb581026f2628 height=29861025
[WARN] [05-16|22:34:12.984] [downloader] Rejected header marked as bad hash=0xcff232a0021469422520093a26ad86a95e8c58fc533c9d97dbceb581026f2628 height=29861025
Erigon version: ./erigon --version
erigon version 2.43.0-dev-065538d5
OS & Version: Windows/Linux/OSX
Ubuntu22.04
Commit hash:
065538d
Erigon Command (with flags/config):
/usr/local/bin/erigon --chain=chapel --datadir=/node/data --metrics --metrics.addr=0.0.0.0 --metrics.port=6060 --private.api.addr=0.0.0.0:9090 --pprof --pprof.addr=0.0.0.0 --pprof.port=6061 --torrent.download.rate=1000mb --http.addr=0.0.0.0 --ws --http.api=eth,erigon,web3,net,debug,trace,txpool --db.size.limit=1000GB --rpc.returndata.limit=900000 --bodies.cache=214748364800 --batchSize=4096M --p2p.protocol=66
Chain/Network:
BSC Tesnet
May 05 09:36:20 bsc-archive-testnet-0-eu erigon[821182]: [WARN] [05-05|09:36:20.161] [downloader] Rejected header marked as bad hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2 height=29517600
May 05 09:36:20 bsc-archive-testnet-0-eu erigon[821182]: [WARN] [05-05|09:36:20.201] [downloader] Rejected header marked as bad hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2 height=29517600
May 05 09:36:20 bsc-archive-testnet-0-eu erigon[821182]: [WARN] [05-05|09:36:20.205] [downloader] Rejected header marked as bad hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2 height=29517600
May 05 09:36:20 bsc-archive-testnet-0-eu erigon[821182]: [WARN] [05-05|09:36:20.275] [downloader] Rejected header marked as bad hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2 height=29517600
May 05 09:36:20 bsc-archive-testnet-0-eu erigon[821182]: [WARN] [05-05|09:36:20.290] [downloader] Rejected header marked as bad hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2 height=29517600
May 05 09:36:20 bsc-archive-testnet-0-eu erigon[821182]: [WARN] [05-05|09:36:20.293] [downloader] Rejected header marked as bad hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2 height=29517600
May 05 09:36:20 bsc-archive-testnet-0-eu erigon[821182]: [WARN] [05-05|09:36:20.352] [downloader] Rejected header marked as bad hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2 height=29517600
May 05 09:36:20 bsc-archive-testnet-0-eu erigon[821182]: [WARN] [05-05|09:36:20.356] [downloader] Rejected header marked as bad hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2 height=29517600
May 05 09:36:20 bsc-archive-testnet-0-eu erigon[821182]: [WARN] [05-05|09:36:20.388] [downloader] Rejected header marked as bad hash=0xdcbeae9b9587cbecf20e21a44940f3ba1a46e8e5509f1be4d8806eb76c9734e2 height=29517600
Tried to unwind the node multiple times and different ranges. The node gets stuck at the same block.
When I synchronized the bsc chain, it was stuck in this position:
The command I used is ./build/bin/erigon --chain bsc --sentry.drop-useless-peers --datadir /media/Disk/chain/bsc/chaindata --torrent.download.rate=20mb --http.api=eth,erigon,web3,net,debug,trace,txpool --rpc.batch.concurrency 10
The version I used is v1.0.8.
I also try to remove the flag --sentry.drop-useless-peers, but it got the same issue.
After plato, verifyVoteAttestation
will call snapshot
, it results a very deep recursive call.
Need to check, if it is reasonable and fix if needed.
Post the call stack here:
0 0x0000000001538910 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:705
1 0x0000000001540be0 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).GetJustifiedNumberAndHash
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:1607
2 0x0000000001536ae0 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:447
3 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
4 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
5 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
6 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
7 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
8 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
9 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
10 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
11 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
12 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
13 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
14 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
15 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
16 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
17 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
18 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
19 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
20 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
21 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
22 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
23 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
24 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
25 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
26 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
27 0x00000000015391b4 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).snapshot
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:791
28 0x0000000001536c44 in github.com/ledgerwatch/erigon/consensus/parlia.(*Parlia).verifyVoteAttestation
at github.com/ledgerwatch/erigon/consensus/parlia/parlia.go:462
NA
Could add a robot to monitor and alert once the Client failed to sync for a while, like: 1 minutes
NA
hi
Thanks for your great work, I switched bsc to this branch, but still can't sync blocks normally, stuck at a block before the hard fork. please help me.
[INFO] [04-12|11:27:45.847] Req/resp stats req=0 reqMin=0 reqMax=0 skel=17 skelMin=27281022 skelMax=27318078 resp=103 respMin=27280978 respMax=27281022 dups=90
[INFO] [04-12|11:28:05.294] [p2p] GoodPeers eth67=1 eth66=25
[INFO] [04-12|11:28:05.811] [txpool] stat pending=11 baseFee=0 queued=22 alloc=2.7GB sys=4.0GB
[INFO] [04-12|11:28:05.848] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:28:05.848] Req/resp stats req=0 reqMin=0 reqMax=0 skel=17 skelMin=27281022 skelMax=27318078 resp=102 respMin=27281022 respMax=27281022 dups=99
[INFO] [04-12|11:28:25.848] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:28:25.848] Req/resp stats req=0 reqMin=0 reqMax=0 skel=17 skelMin=27281022 skelMax=27318078 resp=102 respMin=27281022 respMax=27281022 dups=88
[INFO] [04-12|11:28:45.848] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:28:45.848] Req/resp stats req=0 reqMin=0 reqMax=0 skel=17 skelMin=27281022 skelMax=27318078 resp=102 respMin=27281022 respMax=27281022 dups=95
[INFO] [04-12|11:29:05.812] [txpool] stat pending=11 baseFee=0 queued=24 alloc=3.0GB sys=4.0GB
[INFO] [04-12|11:29:05.847] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:29:05.847] Req/resp stats req=0 reqMin=0 reqMax=0 skel=17 skelMin=27281022 skelMax=27318078 resp=102 respMin=27281022 respMax=27281022 dups=90
[INFO] [04-12|11:29:25.848] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:29:25.848] Req/resp stats req=0 reqMin=0 reqMax=0 skel=17 skelMin=27281022 skelMax=27318078 resp=102 respMin=27281022 respMax=27281022 dups=92
[INFO] [04-12|11:29:45.848] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:29:45.848] Req/resp stats req=0 reqMin=0 reqMax=0 skel=17 skelMin=27281022 skelMax=27318078 resp=102 respMin=27281022 respMax=27281022 dups=88
[INFO] [04-12|11:30:05.294] [p2p] GoodPeers eth66=29 eth67=1
[INFO] [04-12|11:30:05.811] [txpool] stat pending=12 baseFee=0 queued=25 alloc=2.5GB sys=4.0GB
[INFO] [04-12|11:30:05.848] [2/15 Headers] No block headers to write in this log period block number=27281023
[INFO] [04-12|11:30:05.848] Req/resp stats req=0 reqMin=0 reqMax=0 skel=17 skelMin=27281022 skelMax=27318078 resp=106 respMin=27280978 respMax=27281022 dups=94
Erigon version: ./erigon --version
erigon version 2.43.0-dev-7bfcc2bb
OS & Version: Windows/Linux/OSX
Ubuntu22.04
Commit hash:
7bfcc2b
Erigon Command (with flags/config):
/usr/local/bin/erigon --chain=chapel --datadir=/node/data --metrics --metrics.addr=0.0.0.0 --metrics.port=6060 --private.api.addr=0.0.0.0:9090 --pprof --pprof.addr=0.0.0.0 --pprof.port=6061 --torrent.download.rate=1000mb --bodies.cache=214748364800 --batchSize=4096M --p2p.protocol=66 --db.size.limit=1800GB --http=false
Chain/Network:
Chapel
May 12 09:18:58 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:18:58.872] [2/15 Headers] Waiting for headers... from=29614555
May 12 09:18:59 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:18:59.366] [parlia] snapshots build, gather headers block=29600000
May 12 09:18:59 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:18:59.545] New txs subscriber joined
May 12 09:18:59 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:18:59.546] new subscription to newHeaders established
May 12 09:18:59 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:18:59.773] [txpool] Started
May 12 09:19:01 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:19:01.833] local tx propagated tx_hash=f2f37c3bcf91e0a766cd16a900522fbad0c7c47de287ff2f6226f65299f2d4d2 announced to peers=3 broadcast to peers=1 baseFee=0
May 12 09:19:20 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:19:20.345] local tx propagated tx_hash=d4f4f159b12f7639a07d364cf1c176df1fb03380d5eb1f086ed6b828d3c41ae4 announced to peers=3 broadcast to peers=1 baseFee=0
May 12 09:19:58 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:19:58.858] [txpool] stat pending=10000 baseFee=0 queued=29915 alloc=241.6MB sys=277.0MB
May 12 09:20:45 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:20:45.468] local tx propagated tx_hash=c4aa0629eb35a60458200f4ec3808d720d929171206b3da574f587349b203a54 announced to peers=5 broadcast to peers=2 baseFee=0
May 12 09:20:51 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:20:51.566] local tx propagated tx_hash=ff2dcc715ca429e13b639c49135a3b338854586f2641fbd04564565ef4291cdf announced to peers=5 broadcast to peers=2 baseFee=0
May 12 09:20:58 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:20:58.240] [p2p] GoodPeers eth66=5
May 12 09:20:58 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:20:58.885] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=293.0MB sys=331.5MB
May 12 09:20:59 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:20:59.609] local tx propagated tx_hash=98f4632116188036891d9e9ef231e2733f8571f4f1a6a973f003e74e21d4594c announced to peers=4 broadcast to peers=2 baseFee=0
May 12 09:21:58 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:21:58.858] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=236.6MB sys=332.4MB
May 12 09:22:05 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:22:05.772] local tx propagated tx_hash=319f07cf8bd43f2654e07ff76472e44e9769723c14a74856e02c5626e924c6ef announced to peers=4 broadcast to peers=2 baseFee=0
May 12 09:22:47 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:22:47.127] local tx propagated tx_hash=7106493a14acca31e146f7e8852333b06912cb8aa1390a8e23432c5dacb4738c announced to peers=4 broadcast to peers=2 baseFee=0
May 12 09:22:47 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:22:47.397] local tx propagated tx_hash=7106493a14acca31e146f7e8852333b06912cb8aa1390a8e23432c5dacb4738c announced to peers=4 broadcast to peers=2 baseFee=0
May 12 09:22:47 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:22:47.397] local tx propagated tx_hash=a30722dfac82d249fa8481aea50f11f7898f88e36446cd1a89914e8bfdea5b58 announced to peers=4 broadcast to peers=2 baseFee=0
May 12 09:22:58 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:22:58.240] [p2p] GoodPeers eth66=5
May 12 09:22:58 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:22:58.887] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=299.2MB sys=340.6MB
May 12 09:23:58 bsc-archive-testnet-0-eu erigon[870216]: [INFO] [05-12|09:23:58.859] [txpool] stat pending=10000 baseFee=0 queued=30000 alloc=227.3MB sys=344.6MB
Updated to the latest version v1.0.4, worked for 3 days, and after that, the node remained stuck with no errors.
We've already tried restarting the node multiple times and even attempted to unwind it, but no luck.
I think this information is outdated and need to be updated
https://github.com/node-real/bsc-erigon#system-requirements
pls also update the description here:
https://github.com/bnb-chain/bsc-snapshots#usage-1
OS: linux (amd64)
OS Image: Amazon Linux 2
Kernel version: 5.4.226-129.415.amzn2.x86_64
Container runtime: docker://20.10.17
Erigon version: 2.40.0-dev-3da15fcb
Erigon Command (with flags/config):
erigon --chain=bsc --datadir=/data/erigon --config=/home/erigon/config.toml --prune=hrtc --db.pagesize=16k --log.console.verbosity=info --nat=extip:$(POD_IP) --maxpeers=1000 --p2p.protocol=66 --torrent.download.slots=20 --torrent.download.rate=512mb --bodies.cache=214748364800 --batchSize=4096M
Concensus Layer:
Concensus Layer Command (with flags/config):
Chain/Network: bsc / 56
node sync to head and remain synced consistently
The node stopped syncing (here, stucked at bodies stage) and drifted away from chain's head without any changes of configuration.
Also, please note that we already use the config flags mentioned in #41.
The solutions proposed there using the integration
command do not work either.
launch the client, and sync stucks at a fixed block number.
[INFO] [05-04|12:41:20.217] [5/15 Bodies] No block bodies to write in this log period block number=27910849
[INFO] [05-04|12:41:40.217] [5/15 Bodies] Downloading block bodies block_num=27910849 delivery/sec=334.0KB wasted/sec=0B remaining=4935 delivered=264 cache=4.8GB alloc=9.2GB sys=16.2GB
[INFO] [05-04|12:42:00.217] [5/15 Bodies] No block bodies to write in this log period block number=27910849
[INFO] [05-04|12:42:00.221] [5/15 Bodies] DONE in=1m40.005033072s
[INFO] [05-04|12:42:03.978] Commit cycle in=3.756665787s
[INFO] [05-04|12:42:03.978] Timings (slower than 50ms) Headers=6.616s Bodies=1m40.005s
[INFO] [05-04|12:42:03.978] Tables PlainState=233.4GB AccountChangeSet=1.7GB StorageChangeSet=3.6GB BlockTransaction=5.1GB TransactionLog=9.1GB FreeList=45.2MB ReclaimableSpace=181.0GB
[INFO] [05-04|12:42:03.980] RPC Daemon notified of new headers from=27914761 to=27915785 hash=0x684047a81b5a66d11e06e1fc4d5f432d1de4b8a194f1e20c20c9241fe5c98fc0 header sending=1.239498ms log sending=324ns
[INFO] [05-04|12:42:03.981] [2/15 Headers] Waiting for headers... from=27915785
[INFO] [05-04|12:42:06.711] [parlia] snapshots build, gather headers block=27900000
[INFO] [05-04|12:42:06.713] [parlia] snapshots build, recover from headers block=27900000
[INFO] [05-04|12:42:07.850] [parlia] snapshots build, gather headers block=27900000
[INFO] [05-04|12:42:07.852] [parlia] snapshots build, recover from headers block=27900000
[INFO] [05-04|12:42:09.759] [parlia] snapshots build, gather headers block=27900000
[INFO] [05-04|12:42:09.769] [parlia] snapshots build, recover from headers block=27900000
[INFO] [05-04|12:42:09.866] [2/15 Headers] Processed highest inserted=27915821 age=7s
[INFO] [05-04|12:42:09.885] [5/15 Bodies] Processing bodies... from=27910849 to=27915821
[INFO] [05-04|12:42:10.248] [txpool] stat pending=10000 baseFee=268 queued=30000 alloc=6.1GB sys=16.2GB
[INFO] [05-04|12:42:29.886] [5/15 Bodies] Downloading block bodies block_num=27910849 delivery/sec=313.3KB wasted/sec=102.5KB remaining=4971 delivered=103 cache=4.8GB alloc=6.3GB sys=16.2GB
[INFO] [05-04|12:42:49.886] [5/15 Bodies] Downloading block bodies block_num=27910849 delivery/sec=417.2KB wasted/sec=0B remaining=4971 delivered=249 cache=4.8GB alloc=5.8GB sys=16.2GB
[INFO] [05-04|12:43:07.902] [p2p] GoodPeers eth66=708
[INFO] [05-04|12:43:09.885] [5/15 Bodies] No block bodies to write in this log period block number=27910849
Erigon version: ./erigon --version
v1.0.2 3da15fc
OS & Version: Windows/Linux/OSX
Linux
Commit hash:
./build/bin/erigon --datadir="/data/bsc" --chain=bsc --port=30303 --http.port=8545 --authrpc.port=8551 --torrent.port=42069 --private.api.addr=127.0.0.1:9090 --http --ws --http.api=web3,net,eth,debug,trace,txpool --http.corsdomain=* --http.addr=0.0.0.0 --http.vhosts=* --snapshots=false bootnodes=enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311,enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311,enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311 --bodies.cache=214748364800 --batchSize=4096M --db.pagesize=16k --maxpeers=500
Hi,
Thank you for your great work,
In the last week, it has been lagging behind by about 500 blocks, and it was only about 50 blocks behind in the previous week. It feels like it is always chasing blocks, and it has never been completed. I have used erigon for several months, and I remember that it has always been relatively stable. What is the reason for this? Is there any hope of getting this to work?
INFO] [04-23|07:59:24.270] [txpool] stat pending=2378 baseFee=8 queued=530 alloc=3.4GB sys=9.1GB
[INFO] [04-23|07:59:28.827] [2/15 Headers] No block headers to write in this log period block number=27593858
[INFO] [04-23|07:59:48.827] [2/15 Headers] No block headers to write in this log period block number=27593858
[INFO] [04-23|08:00:00.131] [2/15 Headers] Processed highest inserted=27593889 age=52s
[INFO] [04-23|08:00:00.132] [2/15 Headers] DONE in=9m31.304740802s
[INFO] [04-23|08:00:00.423] [5/15 Bodies] Processing bodies... from=27593480 to=27593889
[INFO] [04-23|08:00:20.424] [5/15 Bodies] Downloading block bodies block_num=27593582 delivery/sec=909.8KB wasted/sec=0B remaining=306 delivered=288 cache=11.6MB alloc=3.8GB sys=9.1GB
[INFO] [04-23|08:00:24.268] [txpool] stat pending=2456 baseFee=8 queued=533 alloc=3.8GB sys=9.1GB
[INFO] [04-23|08:00:30.391] [5/15 Bodies] Processed highest=27593889
[INFO] [04-23|08:00:30.391] [6/15 Senders] Started from=27593480 to=27593889
[INFO] [04-23|08:00:31.883] [7/15 Execution] Blocks execution from=27593480 to=27593889
[INFO] [04-23|08:00:51.905] [7/15 Execution] Executed blocks number=27593531 blk/s=2.5 tx/s=447.2 Mgas/s=70.5 gasState=0.00 batch=1.8MB alloc=3.7GB sys=9.1GB
[INFO] [04-23|08:01:12.331] [7/15 Execution] Executed blocks number=27593593 blk/s=3.0 tx/s=527.6 Mgas/s=83.8 gasState=0.01 batch=4.3MB alloc=4.3GB sys=9.1GB
[INFO] [04-23|08:01:23.758] [p2p] GoodPeers eth66=15 eth68=14
[INFO] [04-23|08:01:24.270] [txpool] stat pending=2520 baseFee=9 queued=539 alloc=4.6GB sys=9.1GB
[INFO] [04-23|08:01:31.896] [7/15 Execution] Executed blocks number=27593658 blk/s=3.3 tx/s=602.8 Mgas/s=90.7 gasState=0.01 batch=7.3MB alloc=4.9GB sys=9.1GB
[INFO] [04-23|08:01:52.406] [7/15 Execution] Executed blocks number=27593723 blk/s=3.2 tx/s=568.2 Mgas/s=87.2 gasState=0.01 batch=10.3MB alloc=5.4GB sys=9.1GB
[INFO] [04-23|08:02:11.937] [7/15 Execution] Executed blocks number=27593792 blk/s=3.5 tx/s=628.9 Mgas/s=93.4 gasState=0.02 batch=13.3MB alloc=3.7GB sys=9.1GB
[INFO] [04-23|08:02:27.267] [txpool] stat pending=2590 baseFee=9 queued=544 alloc=4.1GB sys=9.1GB
[INFO] [04-23|08:02:32.043] [7/15 Execution] Executed blocks number=27593868 blk/s=3.8 tx/s=634.1 Mgas/s=88.2 gasState=0.02 batch=16.4MB alloc=4.2GB sys=9.1GB
[INFO] [04-23|08:03:02.403] [7/15 Execution] Completed on block=27593889
[INFO] [04-23|08:03:02.403] [7/15 Execution] DONE in=2m30.519920731s
[INFO] [04-23|08:03:02.403] [8/15 HashState] Promoting plain state from=27593480 to=27593889
[INFO] [04-23|08:03:02.403] [8/15 HashState] Incremental promotion from=27593480 to=27593889 codes=true csbucket=AccountChangeSet
[INFO] [04-23|08:03:20.987] [8/15 HashState] Incremental promotion from=27593480 to=27593889 codes=false csbucket=AccountChangeSet
[INFO] [04-23|08:03:23.758] [p2p] GoodPeers eth66=15 eth68=14
[INFO] [04-23|08:03:24.270] [txpool] stat pending=2684 baseFee=9 queued=559 alloc=5.0GB sys=9.1GB
[INFO] [04-23|08:03:51.794] [8/15 HashState] ETL [2/2] Loading into=HashedAccount current_prefix=ea56aac2
[INFO] [04-23|08:03:54.919] [8/15 HashState] Incremental promotion from=27593480 to=27593889 codes=false csbucket=StorageChangeSet
[INFO] [04-23|08:04:24.268] [txpool] stat pending=2767 baseFee=9 queued=605 alloc=3.5GB sys=9.1GB
[INFO] [04-23|08:04:27.289] [8/15 HashState] ETL [2/2] Loading into=HashedStorage current_prefix=d97dd5b8
[INFO] [04-23|08:04:57.290] [8/15 HashState] ETL [2/2] Loading into=HashedStorage current_prefix=e9dae3d7
[INFO] [04-23|08:05:18.390] [8/15 HashState] DONE in=2m15.987694342s
[INFO] [04-23|08:05:18.391] [9/15 IntermediateHashes] Generating intermediate hashes from=27593480 to=27593889
[INFO] [04-23|08:05:23.757] [p2p] GoodPeers eth66=15 eth68=14
[INFO] [04-23|08:05:24.270] [txpool] stat pending=2818 baseFee=9 queued=670 alloc=4.3GB sys=9.1GB
[INFO] [04-23|08:05:52.250] [9/15 IntermediateHashes] Calculating Merkle root current key=4fef3f1c
[INFO] [04-23|08:06:22.254] [9/15 IntermediateHashes] Calculating Merkle root current key=9e68529e
[INFO] [04-23|08:06:24.268] [txpool] stat pending=3586 baseFee=9 queued=765 alloc=3.9GB sys=9.1GB
[INFO] [04-23|08:07:00.632] [9/15 IntermediateHashes] Calculating Merkle root current key=d97dd5b8
[INFO] [04-23|08:07:23.758] [p2p] GoodPeers eth66=15 eth68=14
[INFO] [04-23|08:07:33.281] [txpool] stat pending=3665 baseFee=9 queued=812 alloc=5.8GB sys=9.1GB
[INFO] [04-23|08:07:41.072] [9/15 IntermediateHashes] Calculating Merkle root current key=e9dae3d7
[INFO] [04-23|08:07:52.505] [9/15 IntermediateHashes] Calculating Merkle root current key=fe1c2c3b
[INFO] [04-23|08:08:14.671] [9/15 IntermediateHashes] DONE in=2m56.280062354s
[INFO] [04-23|08:08:15.242] [10/15 CallTraces] Pruned call trace intermediate table from=27503480 to=27503888
[INFO] [04-23|08:08:24.268] [txpool] stat pending=3757 baseFee=9 queued=854 alloc=4.1GB sys=9.1GB
[INFO] [04-23|08:08:45.332] [10/15 CallTraces] ETL [2/2] Loading into=CallFromIndex current_prefix=bd902132
[INFO] [04-23|08:09:23.758] [p2p] GoodPeers eth68=14 eth66=15
[INFO] [04-23|08:09:24.272] [txpool] stat pending=3838 baseFee=9 queued=906 alloc=4.9GB sys=9.1GB
[INFO] [04-23|08:09:25.567] [10/15 CallTraces] DONE in=1m10.896425372s
[INFO] [04-23|08:09:55.879] [11/15 AccountHistoryIndex] ETL [2/2] Loading into=AccountHistory current_prefix=bf2a2fa0
[INFO] [04-23|08:10:24.268] [txpool] stat pending=3945 baseFee=9 queued=941 alloc=3.9GB sys=9.1GB
[INFO] [04-23|08:10:39.026] [12/15 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=55d39832
[INFO] [04-23|08:11:09.026] [12/15 StorageHistoryIndex] ETL [2/2] Loading into=StorageHistory current_prefix=e3b1d32e
[INFO] [04-23|08:11:15.907] [12/15 StorageHistoryIndex] DONE in=1m8.279033927s
[INFO] [04-23|08:11:15.907] [13/15 LogIndex] processing from=27593481 to=27593889
Hi,
Thanks for your great work!
My test network works well in v1.0.2, but just now when I upgrade v1.0.3, it has the following error, is this related to the machine being ARM64?
err="fail to open mdbx: mdbx_env_open: MDBX_CORRUPTED: Maybe free space is over on disk. Otherwise it's hardware failure. Before creating issue please use tools like https://www.memtest86.com to test RAM and tools like https://www.smartmontools.org to test Disk. To handle hardware risks: use ECC RAM, use RAID of disks, run multiple application instances (or do backups). If hardware checks passed - check FS settings - 'fsync' and 'flock' must be enabled. Otherwise - please create issue in Application repo. On default DURABLE mode, power outage can't cause this error. On other modes - power outage may break last transaction and mdbx_chk can recover db in this case, see '-t' and '-0|1|2' options., label: consensus, trace: [kv_mdbx.go:266 kv_mdbx.go:397 db.go:20 config.go:75 backend.go:455 node.go:112 main.go:59 command.go:274 app.go:332 app.go:309 main.go:36 proc.go:250 asm_arm64.s:1172]" stack="[main.go:31 panic.go:884 kv_mdbx.go:399 db.go:20 config.go:75 backend.go:455 node.go:112 main.go:59 command.go:274 app.go:332 app.go:309 main.go:36 proc.go:250 asm_arm64.s:1172]"
Erigon version: ./erigon --version
erigon version 2.40.0-dev
OS & Version: Windows/Linux/OSX
Linux
Commit hash:
Erigon Command (with flags/config):
erigon --chain=bsc --datadir=/srv/svc --metrics --metrics.addr=0.0.0.0 --metrics.port=6060 --private.api.addr=0.0.0.0:9090 --pprof --pprof.addr=0.0.0.0 --pprof.port=6061 --http.api=eth,erigon,web3,net,debug,trace,txpool,parity,bor --prune htrc --ws
Concensus Layer:
Concensus Layer Command (with flags/config):
Chain/Network: 56 Mainnet
it should sync to the currentblock
it is lagging a few blocks to sync
just download and sync
[backtrace]
Hello there,
My nodereal erigon build is lagging to sync for about 2 weeks now.
once in every few blocks sync it displays like this
[INFO] [04-11|03:35:14.497] [5/15 Bodies] No block bodies to write in this log period block number=27241889
[INFO] [04-11|03:35:34.498] [5/15 Bodies] No block bodies to write in this log period block number=27241889
and waits for a few seconds and catchup to the current block
and then it occurs again.
it is delaying
The data volume of BSC Testnet is becoming more and more large, it could takes days for people to sync from genesis.
It would be better to provide one in: https://github.com/bnb-chain/bsc-snapshots#erigon-bsc-snapshot
No need to update frequently, quarterly update could be ok IMO
Erigon version: v1.0.7
Should not print so many error logs.
Saw lots of error message after Plato:
[EROR] [05-17|08:44:46.953] GetJustifiedNumberAndHash snapshot error="unknown ancestor" blockNumber=29871616 blockHash=0xe149bffe9da5aa36e41181a709fee5df2fa00c0875ebc55618e19fb6c1d887d2
[EROR] [05-17|08:44:46.953] GetJustifiedNumberAndHash snapshot error="unknown ancestor" blockNumber=29871616 blockHash=0xe149bffe9da5aa36e41181a709fee5df2fa00c0875ebc55618e19fb6c1d887d2
[EROR] [05-17|08:44:46.958] GetJustifiedNumberAndHash snapshot error="unknown ancestor" blockNumber=29871617 blockHash=0x51c188a8df9a6537a82bb417e58a80e3cd3fc4c8a841a0de15da6edc83191d29
Need to check if these logs are reasonable or not.
It can be easily reproduced, when the node reached the Plato hard fork height.
NA
Erigon version: ./erigon --version
OS & Version: Windows/Linux/OSX
Commit hash:
Erigon Command (with flags/config):
Concensus Layer:
Concensus Layer Command (with flags/config):
Chain/Network: Chapel
the code of address 0x0000000000000000000000000000000000001000 has changed after Luban hardfork (0x1bf01ca), but erigon did not update the code. but if getCode from "latest", the result is correct.
request:
{ "jsonrpc": "2.0", "id": 1, "method": "eth_getCode", "params": [ "0x0000000000000000000000000000000000001000", "0x1bf01cc" ] }
expected:
{ "jsonrpc": "2.0", "id": 1, "result": "0x60806040526004361061046c57600.........." }
{ "jsonrpc": "2.0", "id": 1, "result": "0x60806040526004361061040557..........." }
[backtrace]
This should only be used in very rare cases e.g. if you are not 100% sure if something is a bug or asking a question that leads to improving the documentation. For general questions please use Erigon's discord.
Hi Team,
I have started the node using snapshots & it syncs really slow only 7-8 blk/sec. Here are the hardware details for more information:
Machine SKU: Standard_L16
vCPU: 16
RAM: 128GB
I am running the service with the following command:
exec erigon --chain bsc --snapshots=true --db.pagesize=16k --datadir=/bnb-backup/data --txpool.disable --rpc.batch.concurrency=1500 --rpc.batch.limit=1500 --torrent.upload.rate=512mb --torrent.download.rate=512mb --http.addr=0.0.0.0 --http.port=8545 --rpc.returndata.limit=1024000 --p2p.protocol=66
that way it takes infinite time to sync with the bsc mainnet.
Our servers are all centos7, and the executable file packaged by release lacks relevant dependencies on my machine:
./erigon: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ./erigon)
./erigon: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by ./erigo
I don't want to make any destructive changes to my system.
Thanks~
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.