Code Monkey home page Code Monkey logo

copernicus's People

Contributors

antoniowang0526 avatar detailyang avatar enrique8191 avatar ericnes avatar fffreedom avatar fingera avatar georgefff avatar ggcodemasta avatar hycxa avatar intelwin avatar jjz avatar ludete avatar machine-bot avatar mazeyu avatar qiwei9743 avatar qshuai avatar radar3 avatar terrencege avatar unixisevil avatar whitefen avatar whunmr avatar wolfstudy avatar wonghoifung avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

copernicus's Issues

debug log may cause the data race

Hello.
I'm not sure it should be fixed or not. The debug log may cause the data race as the following or syncmanager.go#L1078

	log.Debug(
		"invBlkCnt=%d len(invVects)=%d sm.requestBlkInv=%v  peer=%p(%#v) sm.syncPeer=%p",
		invBlkCnt, len(invVects), sm.requestBlkInvCnt, peer, peer, sm.syncPeer)

The verbs %#v will case the peer's fields data race because of the reflect
after receiving inv message and you can easily reproduce it when build with go build -race

Try to read not exists block data during activateBestChain

Describe the bug
lots logs seems try to read block at invalid position (File=-1, Pos=0)

To Reproduce
running latest code in the master

Expected behavior
not try to read block data if block data has has not been downloaded yet.

Additional context

2018/12/05 10:01:41.147 [E] [disk.go:229]  ReadBlockFromDisk: OpenBlockFile failed for BlcokDiskPos(File=-1, Pos=0)
2018/12/05 10:01:41.147 [E] [lchain.go:242]  error: FailedToReadBlock: false
2018/12/05 10:01:41.147 [E] [blockservice.go:99]   ActivateBestChain failed :module: disk, errcode: 5006: FailedToReadBlock
2018/12/05 10:01:41.147 [E] [syncmanager.go:619]  ProcessBlockCallBack err:module: disk, errcode: 5006: FailedToReadBlock, hash: 0000000000000000297a83beef2cf9bffd83fb9080c1a5236d1cb1746ca54c0e
2018/12/05 10:01:41.254 [E] [disk.go:229]  ReadBlockFromDisk: OpenBlockFile failed for BlcokDiskPos(File=-1, Pos=0)
2018/12/05 10:01:41.254 [E] [lchain.go:242]  error: FailedToReadBlock: false
2018/12/05 10:01:41.254 [E] [blockservice.go:99]   ActivateBestChain failed :module: disk, errcode: 5006: FailedToReadBlock
2018/12/05 10:01:41.254 [E] [syncmanager.go:619]  ProcessBlockCallBack err:module: disk, errcode: 5006: FailedToReadBlock, hash: 00000000000000007c2a14250b43a28843f1c301bbbc4e4af6aa25cc8942cf0c
2018/12/05 10:01:41.328 [E] [disk.go:229]  ReadBlockFromDisk: OpenBlockFile failed for BlcokDiskPos(File=-1, Pos=0)
2018/12/05 10:01:41.328 [E] [lchain.go:242]  error: FailedToReadBlock: false
2018/12/05 10:01:41.328 [E] [blockservice.go:99]   ActivateBestChain failed :module: disk, errcode: 5006: FailedToReadBlock

BlockUndo UnSerialize: the txUndo UnSerialize error: EOF

Describe the bug
when running undo testcases, BlockUndo UnSerialize will failure with EOF

To Reproduce
Steps to reproduce the behavior:

  1. running testcase single_node_undo.py
pipenv run python single_node_undo.py --tracerpc -l DEBUG --nocleanup

Expected behavior
undo successfully and no undo error in copernicus.log

Additional context
error logs:

2018/10/19 12:19:52.287 [D] [peer.go:515]  Updating last block height of peer 127.0.0.1:64876 from 101 to 102
2018/10/19 12:19:52.287 [D] [syncmanager.go:695]  len of Requested block:0, requestBlkInvCnt: 0
2018/10/19 12:19:53.291 [D] [message.go:408]  begin Decode msg ....
2018/10/19 12:19:53.291 [D] [block.go:77]  block Unserialize.... 
2018/10/19 12:19:53.292 [D] [peer.go:1206]  read message: Received block (hash 49472972d4d31047c0d51f490e6f8a0b9b96d5c321d96d066b18b7601bfce278, ver 1, 100 tx, 1539922892) from 127.0.0.1:64876 (inbound)
2018/10/19 12:19:53.292 [I] [peer.go:1563]  Read message *wire.MsgBlock inHandle
2018/10/19 12:19:53.292 [D] [blockservice.go:38]  Begin processing block: 49472972d4d31047c0d51f490e6f8a0b9b96d5c321d96d066b18b7601bfce278, Global Chain height: 102, tipHash: 0b6f0b27e1635c7443054b241c06692f53ee46ed0895f3a67055b998c1f8a376, coinsTip hash: 0b6f0b27e1635c7443054b241c06692f53ee46ed0895f3a67055b998c1f8a376
2018/10/19 12:19:53.293 [D] [block.go:35]  block Serialize....
2018/10/19 12:19:53.294 [D] [chain.go:539]  AddToIndexMap:49472972d4d31047c0d51f490e6f8a0b9b96d5c321d96d066b18b7601bfce278 index height:102
2018/10/19 12:19:53.294 [D] [block.go:35]  block Serialize....
2018/10/19 12:19:53.296 [D] [lblock.go:55]  block(hash: 49472972d4d31047c0d51f490e6f8a0b9b96d5c321d96d066b18b7601bfce278) data is written to disk
2018/10/19 12:19:53.297 [D] [block.go:77]  block Unserialize.... 
2018/10/19 12:19:53.298 [D] [disk.go:192]  UndoReadFromDisk: read undo file size: 38 and num: 38.
2018/10/19 12:19:53.298 [E] [undo.go:54]  TxUndo UnSerialize: the read count is: 0, error: EOF
2018/10/19 12:19:53.298 [E] [undo.go:118]  BlockUndo UnSerialize: the txUndo UnSerialize error: EOF
2018/10/19 12:19:53.298 [E] [disk.go:203]  UndoReadFromDisk: unserialize error: EOF, pos is: BlcokDiskPos(File=0, Pos=3737), undoData is: [99 1 5 50 7 81]
2018/10/19 12:19:53.299 [E] [lchain.go:496]  DisconnectBlock(): reading undo data failed, pos is: BlcokDiskPos(File=0, Pos=3737), block undo is: &{txundo:[]}
2018/10/19 12:19:53.299 [E] [lchain.go:326]  DisconnectTip(): DisconnectBlock 0b6f0b27e1635c7443054b241c06692f53ee46ed0895f3a67055b998c1f8a376 failed 
2018/10/19 12:19:53.299 [E] [blockservice.go:90]   ActivateBestChain failed :module: disk, global errcode: 5007,  desc: DisconnectTipUndoFailed
2018/10/19 12:19:53.299 [D] [blockservice.go:46]  processBlock failed ...
2018/10/19 12:19:53.299 [D] [syncmanager.go:632]  len of Requested block:0, sm.requestBlockInvCnt: 0
2018/10/19 12:19:53.299 [D] [syncmanager.go:650]  ProcessBlockCallBack err:module: disk, global errcode: 5007,  desc: DisconnectTipUndoFailed
2018/10/19 12:19:53.299 [D] [message.go:408]  begin Decode msg ....
2018/10/19 12:19:53.299 [D] [block.go:77]  block Unserialize.... 
2018/10/19 12:19:53.300 [D] [peer.go:1206]  read message: Received block (hash 783f75871e9f3f2f00f37073ebe0eb5883c5f55fa661cba61dafbffcf61d64a3, ver 1, 100 tx, 1539922893) from 127.0.0.1:64876 (inbound)
2018/10/19 12:19:53.300 [I] [peer.go:1563]  Read message *wire.MsgBlock inHandle
2018/10/19 12:19:53.301 [D] [blockservice.go:38]  Begin processing block: 783f75871e9f3f2f00f37073ebe0eb5883c5f55fa661cba61dafbffcf61d64a3, Global Chain height: 102, tipHash: 0b6f0b27e1635c7443054b241c06692f53ee46ed0895f3a67055b998c1f8a376, coinsTip hash: 0b6f0b27e1635c7443054b241c06692f53ee46ed0895f3a67055b998c1f8a376
2018/10/19 12:19:53.301 [D] [block.go:35]  block Serialize....
2018/10/19 12:19:53.302 [D] [chain.go:539]  AddToIndexMap:783f75871e9f3f2f00f37073ebe0eb5883c5f55fa661cba61dafbffcf61d64a3 index height:103
2018/10/19 12:19:53.302 [D] [block.go:35]  block Serialize....
2018/10/19 12:19:53.303 [D] [lblock.go:55]  block(hash: 783f75871e9f3f2f00f37073ebe0eb5883c5f55fa661cba61dafbffcf61d64a3) data is written to disk
2018/10/19 12:19:53.304 [D] [block.go:77]  block Unserialize.... 
2018/10/19 12:19:53.304 [D] [disk.go:192]  UndoReadFromDisk: read undo file size: 38 and num: 38.
2018/10/19 12:19:53.304 [E] [undo.go:54]  TxUndo UnSerialize: the read count is: 0, error: EOF
2018/10/19 12:19:53.304 [E] [undo.go:118]  BlockUndo UnSerialize: the txUndo UnSerialize error: EOF
2018/10/19 12:19:53.304 [E] [disk.go:203]  UndoReadFromDisk: unserialize error: EOF, pos is: BlcokDiskPos(File=0, Pos=3737), undoData is: [99 1 5 50 7 81]
2018/10/19 12:19:53.305 [E] [lchain.go:496]  DisconnectBlock(): reading undo data failed, pos is: BlcokDiskPos(File=0, Pos=3737), block undo is: &{txundo:[]}
2018/10/19 12:19:53.305 [E] [lchain.go:326]  DisconnectTip(): DisconnectBlock 0b6f0b27e1635c7443054b241c06692f53ee46ed0895f3a67055b998c1f8a376 failed 
2018/10/19 12:19:53.305 [E] [blockservice.go:90]   ActivateBestChain failed :module: disk, global errcode: 5007,  desc: 2018
DisconnectTipUndoFailed/10/19 12:19:53.305 [D] [blockservice.go:46]  processBlock failed ...

concurrent map read and map write: chain.(*Chain).FindBlockIndex

Describe the bug

stderr:
fatal error: concurrent map read and map write

goroutine 24 [running]:
runtime.throw(0x49906fd, 0x21)
	/usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc000c3da80 sp=0xc000c3da50 pc=0x4032862
runtime.mapaccess2(0x48ab040, 0xc0001b2600, 0xc000c3db20, 0x58d0b60, 0x32)
	/usr/local/go/src/runtime/map.go:453 +0x2d0 fp=0xc000c3daf0 sp=0xc000c3da80 pc=0x4010ba0
github.com/copernet/copernicus/model/chain.(*Chain).FindBlockIndex(0xc0000a20a0, 0xee19af15d174fe09, 0x7270feb6e5edb0cf, 0x40b79dc8f9ad30b9, 0x6dff9e1b0b95c134, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/model/chain/chain.go:116 +0x7a fp=0xc000c3db68 sp=0xc000c3daf0 pc=0x464ff2a
github.com/copernet/copernicus/net/server.(*Server).pushBlockMsg(0xc000246540, 0xc000254120, 0xc001c881fc, 0xc0000852c0, 0x0, 0x1, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:1058 +0x82 fp=0xc000c3dc98 sp=0xc000c3db68 pc=0x47b4ce2
github.com/copernet/copernicus/net/server.(*serverPeer).OnGetData(0xc000254120, 0xc0000bc300, 0xc0001d1200)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:619 +0x378 fp=0xc000c3dde0 sp=0xc000c3dc98 pc=0x47b2b18
github.com/copernet/copernicus/net/server.(*serverPeer).OnGetData-fm(0xc0000bc300, 0xc0001d1200)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:1593 +0x43 fp=0xc000c3de10 sp=0xc000c3dde0 pc=0x47c2833
github.com/copernet/copernicus/net/syncmanager.(*SyncManager).messagesHandler(0xc000144210)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:1198 +0x9e7 fp=0xc000c3dfd8 sp=0xc000c3de10 pc=0x4780c47
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000c3dfe0 sp=0xc000c3dfd8 pc=0x40648b1
created by github.com/copernet/copernicus/net/syncmanager.(*SyncManager).Start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:1430 +0xb8

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context

stderr:
fatal error: concurrent map read and map write

goroutine 24 [running]:
runtime.throw(0x49906fd, 0x21)
	/usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc000c3da80 sp=0xc000c3da50 pc=0x4032862
runtime.mapaccess2(0x48ab040, 0xc0001b2600, 0xc000c3db20, 0x58d0b60, 0x32)
	/usr/local/go/src/runtime/map.go:453 +0x2d0 fp=0xc000c3daf0 sp=0xc000c3da80 pc=0x4010ba0
github.com/copernet/copernicus/model/chain.(*Chain).FindBlockIndex(0xc0000a20a0, 0xee19af15d174fe09, 0x7270feb6e5edb0cf, 0x40b79dc8f9ad30b9, 0x6dff9e1b0b95c134, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/model/chain/chain.go:116 +0x7a fp=0xc000c3db68 sp=0xc000c3daf0 pc=0x464ff2a
github.com/copernet/copernicus/net/server.(*Server).pushBlockMsg(0xc000246540, 0xc000254120, 0xc001c881fc, 0xc0000852c0, 0x0, 0x1, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:1058 +0x82 fp=0xc000c3dc98 sp=0xc000c3db68 pc=0x47b4ce2
github.com/copernet/copernicus/net/server.(*serverPeer).OnGetData(0xc000254120, 0xc0000bc300, 0xc0001d1200)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:619 +0x378 fp=0xc000c3dde0 sp=0xc000c3dc98 pc=0x47b2b18
github.com/copernet/copernicus/net/server.(*serverPeer).OnGetData-fm(0xc0000bc300, 0xc0001d1200)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:1593 +0x43 fp=0xc000c3de10 sp=0xc000c3dde0 pc=0x47c2833
github.com/copernet/copernicus/net/syncmanager.(*SyncManager).messagesHandler(0xc000144210)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:1198 +0x9e7 fp=0xc000c3dfd8 sp=0xc000c3de10 pc=0x4780c47
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000c3dfe0 sp=0xc000c3dfd8 pc=0x40648b1
created by github.com/copernet/copernicus/net/syncmanager.(*SyncManager).Start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:1430 +0xb8

goroutine 1 [chan receive]:
main.bchMain(0x4aab960, 0xc0000260a0, 0xc00001e1e0, 0x6, 0x6, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/main.go:82 +0x276
main.main()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/main.go:106 +0x1ed

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:139 +0x9f
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x26
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 7 [chan receive]:
github.com/copernet/copernicus/vendor/github.com/astaxie/beego/logs.(*fileLogWriter).dailyRotate(0xc000224d80, 0xbef167dec3f62ce8, 0xbcf3ae, 0x4fa4760)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/astaxie/beego/logs/file.go:237 +0x240
created by github.com/copernet/copernicus/vendor/github.com/astaxie/beego/logs.(*fileLogWriter).initFd
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/astaxie/beego/logs/file.go:221 +0x599

goroutine 8 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc000246000)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x12c
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x2d1

goroutine 11 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0002049c0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x123
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x686

goroutine 12 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0002049c0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x105
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:143 +0x6a8

goroutine 13 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0002049c0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:834 +0xbad
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:149 +0x898

goroutine 14 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0002049c0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:762 +0x13c
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:150 +0x8ba

goroutine 15 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc0002461c0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x12c
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x2d1

goroutine 16 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc00009a4e0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x123
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x686

goroutine 50 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc00009a4e0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x105
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:143 +0x6a8

goroutine 51 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc00009a4e0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:834 +0xbad
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:149 +0x898

goroutine 52 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc00009a4e0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:762 +0x13c
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:150 +0x8ba

goroutine 36 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc000246380)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x12c
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x2d1

goroutine 54 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc00009a820)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x123
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:142 +0x686

goroutine 55 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc00009a820)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101 +0x105
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:143 +0x6a8

goroutine 56 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc00009a820)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:834 +0xbad
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:149 +0x898

goroutine 57 [select]:
github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc00009a820)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:762 +0x13c
created by github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
	/Users/zhangyong/go/src/github.com/copernet/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:150 +0x8ba

goroutine 58 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 59 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 60 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 61 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 62 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 63 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 64 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 65 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 82 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 83 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 84 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 85 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 86 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 87 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 88 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 89 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 90 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 91 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 92 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 93 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 94 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 95 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 96 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 97 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 98 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 99 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 100 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 101 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 102 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 103 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 104 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 105 [chan receive]:
github.com/copernet/copernicus/logic/ltx.checkScript()
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:659 +0x8b
created by github.com/copernet/copernicus/logic/ltx.ScriptVerifyInit
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/ltx/ltx.go:72 +0xf9

goroutine 107 [select]:
main.interruptListener.func1(0xc00012a080, 0xc000084ae0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/signal.go:33 +0x12a
created by main.interruptListener
	/Users/zhangyong/go/src/github.com/copernet/copernicus/signal.go:28 +0xa1

goroutine 23 [select]:
github.com/copernet/copernicus/net/addrmgr.(*AddrManager).addressHandler(0xc000039400)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/addrmgr/addrmanager.go:360 +0x119
created by github.com/copernet/copernicus/net/addrmgr.(*AddrManager).Start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/addrmgr/addrmanager.go:569 +0xcb

goroutine 37 [IO wait]:
internal/poll.runtime_pollWait(0x5818310, 0x72, 0x16)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc0000c8118, 0x72, 0x4f61800, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc0000c8118, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Accept(0xc0000c8100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x383
net.(*netFD).accept(0xc0000c8100, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x7d
net.(*TCPListener).accept(0xc00000e038, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x56
net.(*TCPListener).Accept(0xc00000e038, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock.go:260 +0x7e
net/http.(*Server).Serve(0xc0001fe5b0, 0x4aab060, 0xc00000e038, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2826 +0x350
github.com/copernet/copernicus/rpc.(*Server).Start.func2(0xc0001fe5b0, 0xc0001fe410, 0x4aab060, 0xc00000e038)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:454 +0x149
created by github.com/copernet/copernicus/rpc.(*Server).Start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:452 +0x2aa

goroutine 38 [IO wait]:
internal/poll.runtime_pollWait(0x5818240, 0x72, 0x17)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc0000c8198, 0x72, 0x4f61800, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc0000c8198, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Accept(0xc0000c8180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x383
net.(*netFD).accept(0xc0000c8180, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x7d
net.(*TCPListener).accept(0xc00000e050, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x56
net.(*TCPListener).Accept(0xc00000e050, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock.go:260 +0x7e
net/http.(*Server).Serve(0xc0001fe5b0, 0x4aab060, 0xc00000e050, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2826 +0x350
github.com/copernet/copernicus/rpc.(*Server).Start.func2(0xc0001fe5b0, 0xc0001fe410, 0x4aab060, 0xc00000e050)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:454 +0x149
created by github.com/copernet/copernicus/rpc.(*Server).Start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:452 +0x2aa

goroutine 39 [select]:
github.com/copernet/copernicus/net/server.(*MsgHandle).startProcess(0xc0001882d0, 0x4aab960, 0xc0000260a8)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/msghandle.go:44 +0x125
created by github.com/copernet/copernicus/net/server.SetMsgHandle
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/msghandle.go:36 +0xaf

goroutine 40 [select]:
github.com/copernet/copernicus/net/server.(*Server).cycle(0xc000246540)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:1749 +0x5b3
created by github.com/copernet/copernicus/net/server.(*Server).Start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:1986 +0x104

goroutine 41 [chan receive]:
main.bchMain.func3(0xc0001fe410)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/main.go:79 +0x47
created by main.bchMain
	/Users/zhangyong/go/src/github.com/copernet/copernicus/main.go:78 +0x25f

goroutine 26 [select]:
github.com/copernet/copernicus/net/connmgr.(*ConnManager).connHandler(0xc000254000)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/connmgr/connmanager.go:225 +0x1ac
created by github.com/copernet/copernicus/net/connmgr.(*ConnManager).Start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/connmgr/connmanager.go:370 +0xd5

goroutine 27 [IO wait]:
internal/poll.runtime_pollWait(0x58184b0, 0x72, 0x14)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc0000c8018, 0x72, 0x4f61800, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc0000c8018, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Accept(0xc0000c8000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x383
net.(*netFD).accept(0xc0000c8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x7d
net.(*TCPListener).accept(0xc00000e028, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x56
net.(*TCPListener).Accept(0xc00000e028, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock.go:260 +0x7e
github.com/copernet/copernicus/net/connmgr.(*ConnManager).listenHandler(0xc000254000, 0x4aab960, 0xc0000260a8, 0x4aab060, 0xc00000e028)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/connmgr/connmanager.go:346 +0x17d
created by github.com/copernet/copernicus/net/connmgr.(*ConnManager).Start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/connmgr/connmanager.go:377 +0x1e4

goroutine 28 [IO wait]:
internal/poll.runtime_pollWait(0x58183e0, 0x72, 0x15)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc0000c8098, 0x72, 0x4f61800, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc0000c8098, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Accept(0xc0000c8080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x383
net.(*netFD).accept(0xc0000c8080, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x7d
net.(*TCPListener).accept(0xc00000e030, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x56
net.(*TCPListener).Accept(0xc00000e030, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/tcpsock.go:260 +0x7e
github.com/copernet/copernicus/net/connmgr.(*ConnManager).listenHandler(0xc000254000, 0x4aab960, 0xc0000260a8, 0x4aab060, 0xc00000e030)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/connmgr/connmanager.go:346 +0x17d
created by github.com/copernet/copernicus/net/connmgr.(*ConnManager).Start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/connmgr/connmanager.go:377 +0x1e4

goroutine 48 [select]:
github.com/copernet/copernicus/peer.(*Peer).stallHandler(0xc0000bc300)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:1396 +0x27e
created by github.com/copernet/copernicus/peer.(*Peer).start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:2001 +0x4e6

goroutine 130 [runnable]:
github.com/copernet/copernicus/peer.(*Peer).queueHandler(0xc0000bc300)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:1633 +0x2ad
created by github.com/copernet/copernicus/peer.(*Peer).start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:2003 +0x53d

goroutine 49 [chan receive]:
github.com/copernet/copernicus/peer.(*Peer).inHandler(0xc0000bc300, 0xc00001eb40)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:1585 +0x9d6
created by github.com/copernet/copernicus/peer.(*Peer).start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:2002 +0x518

goroutine 112 [chan receive]:
github.com/copernet/copernicus/peer.(*Peer).WaitForDisconnect(0xc0000bc300)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:2015 +0x3d
github.com/copernet/copernicus/net/server.(*Server).peerDoneHandler(0xc000246540, 0xc000254120)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:1696 +0x46
created by github.com/copernet/copernicus/net/server.(*Server).inboundPeerConnected
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:1636 +0x1b7

goroutine 131 [runnable]:
time.Time.Unix(0xbef167e3db584a38, 0x4c036b2cb, 0x4fa4760, 0x1)
	/usr/local/go/src/time/time.go:1111 +0x52
github.com/copernet/copernicus/peer.(*Peer).outHandler(0xc0000bc300)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:1791 +0x6af
created by github.com/copernet/copernicus/peer.(*Peer).start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:2004 +0x562

goroutine 132 [select]:
github.com/copernet/copernicus/peer.(*Peer).pingHandler(0xc0000bc300)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:1831 +0x12e
created by github.com/copernet/copernicus/peer.(*Peer).start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:2005 +0x587

goroutine 67 [runnable]:
syscall.Syscall(0x152, 0xc002697e00, 0xc001c4aed8, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/asm_darwin_amd64.s:16 +0x5
syscall.Stat(0xc002697d80, 0x7c, 0xc001c4aed8, 0x0, 0x0)
	/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:1192 +0x104
os.statNolog(0xc002697d80, 0x7c, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/stat_unix.go:31 +0xa5
os.Stat(0xc002697d80, 0x7c, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/stat.go:13 +0x83
os.MkdirAll(0xc002697d80, 0x7c, 0x1ff, 0x0, 0x0)
	/usr/local/go/src/os/path.go:21 +0x71
github.com/copernet/copernicus/persist/disk.OpenDiskFile(0x13cad00000000, 0x4978811, 0x3, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/persist/disk/disk.go:62 +0xb0
github.com/copernet/copernicus/persist/disk.OpenBlockFile(0xc00269b0f0, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/persist/disk/disk.go:50 +0x57
github.com/copernet/copernicus/persist/disk.WriteBlockToDisk(0xc001c760b0, 0xc00269b0f0, 0x5be52800)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/persist/disk/disk.go:282 +0x50
github.com/copernet/copernicus/logic/lblock.WriteBlockToDisk(0xc0026ea000, 0xc001c760b0, 0x0, 0x0, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:49 +0x198
github.com/copernet/copernicus/logic/lblock.AcceptBlock(0xc001c760b0, 0x49be201, 0x0, 0xc00135f16d, 0xc0026ea000, 0x0, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:269 +0x3ce
github.com/copernet/copernicus/service.ProcessNewBlock(0xc001c760b0, 0xc00135f101, 0xc00135f16d, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/service/blockservice.go:80 +0x223
github.com/copernet/copernicus/rpc.generateBlocks(0xc00006aa00, 0x4e2, 0xf3ec7, 0xc0001d2380, 0x0, 0x0, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/mining.go:602 +0x39a
github.com/copernet/copernicus/rpc.handleGenerate(0xc0001fe410, 0x485d000, 0xc000058700, 0xc000200600, 0x0, 0x0, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/mining.go:555 +0x4dc
github.com/copernet/copernicus/rpc.(*Server).standardCmdResult(0xc0001fe410, 0xc00006a9c0, 0xc000200600, 0x0, 0x0, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:218 +0x115
github.com/copernet/copernicus/rpc.(*Server).jsonRPCRead(0xc0001fe410, 0x4aab2e0, 0xc00137a0e0, 0xc0000cc300, 0x1)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:378 +0x114a
github.com/copernet/copernicus/rpc.(*Server).Start.func1(0x4aab2e0, 0xc00137a0e0, 0xc0000cc300)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:447 +0x267
net/http.HandlerFunc.ServeHTTP(0xc0001882c0, 0x4aab2e0, 0xc00137a0e0, 0xc0000cc300)
	/usr/local/go/src/net/http/server.go:1964 +0x44
net/http.(*ServeMux).ServeHTTP(0xc00020d6e0, 0x4aab2e0, 0xc00137a0e0, 0xc0000cc300)
	/usr/local/go/src/net/http/server.go:2361 +0x1a4
net/http.serverHandler.ServeHTTP(0xc0001fe5b0, 0x4aab2e0, 0xc00137a0e0, 0xc0000cc300)
	/usr/local/go/src/net/http/server.go:2741 +0x1e9
net/http.(*conn).serve(0xc0001a6140, 0x4aab920, 0xc00006a340)
	/usr/local/go/src/net/http/server.go:1847 +0x1146
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x7b0

goroutine 69 [IO wait]:
internal/poll.runtime_pollWait(0x58180a0, 0x72, 0x23)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc0001ce218, 0x72, 0x4aa7a00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x96
internal/poll.(*pollDesc).waitRead(0xc0001ce218, 0xc00012b100, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x45
internal/poll.(*FD).Read(0xc0001ce200, 0xc00012b1d4, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x36a
net.(*netFD).Read(0xc0001ce200, 0xc00012b1d4, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x7a
net.(*conn).Read(0xc0001c8010, 0xc00012b1d4, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0xb0
github.com/copernet/copernicus/rpc.(*Server).jsonRPCRead.func1(0x4aae120, 0xc0001c8010, 0xc000200600)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:357 +0xa3
created by github.com/copernet/copernicus/rpc.(*Server).jsonRPCRead
	/Users/zhangyong/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:356 +0x109e

error: concurrent map iteration and map write

Smartphone (please complete the following information):

  • OS: mac os
  • Browser :chrome
  • Version [new master]

Context

The code is the latest code of the current copernicus. When I use ./copernicus --testnet to start, everything ok. At this time, when I use the exporter of prometheus to get the RPC interface of the copernicus client, the following error occurs:

Current data dir: /Users/xiaolong.ran/Library/Application Support/Bitcoincash/testnet
Profile server listening on localhost:6060

fatal error: concurrent map iteration and map write

For detailed error information, please download this link below:

coper_error.log

when i use go build -race, continue to repeat the above steps and try to reproduce the problem, the error log:

coper_race_error.log

Important logs are as follows:

==================
WARNING: DATA RACE
Read at 0x00c000214b70 by goroutine 429:
  github.com/copernet/copernicus/persist/disk.FlushStateToDisk()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/persist/disk/disk.go:397 +0x85f
  github.com/copernet/copernicus/rpc.handleGetTxoutSetInfo()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/rpc/rpcblockchain.go:755 +0x55
  github.com/copernet/copernicus/rpc.(*Server).standardCmdResult()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:217 +0x13b
  github.com/copernet/copernicus/rpc.(*Server).jsonRPCRead()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:378 +0xbdb
  github.com/copernet/copernicus/rpc.(*Server).Start.func1()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:448 +0x212
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/Cellar/go/1.11.2/libexec/src/net/http/server.go:1964 +0x51
  net/http.(*ServeMux).ServeHTTP()
      /usr/local/Cellar/go/1.11.2/libexec/src/net/http/server.go:2361 +0x191
  net/http.serverHandler.ServeHTTP()
      /usr/local/Cellar/go/1.11.2/libexec/src/net/http/server.go:2741 +0xc4
  net/http.(*conn).serve()
      /usr/local/Cellar/go/1.11.2/libexec/src/net/http/server.go:1847 +0x80a

Previous write at 0x00c000214b70 by goroutine 70:
  runtime.mapassign()
      /usr/local/Cellar/go/1.11.2/libexec/src/runtime/map.go:549 +0x0
  github.com/copernet/copernicus/persist.(*PersistGlobal).AddDirtyBlockIndex()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/persist/global.go:56 +0x84
  github.com/copernet/copernicus/model/chain.(*Chain).AddToIndexMap()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/model/chain/chain.go:594 +0x409
  github.com/copernet/copernicus/logic/lblock.AcceptBlockHeader()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:311 +0x5d7
  github.com/copernet/copernicus/service.ProcessBlockHeader()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/service/blockservice.go:19 +0x135
  github.com/copernet/copernicus/net/syncmanager.(*SyncManager).handleHeadersMsg()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:946 +0x52d
  github.com/copernet/copernicus/net/syncmanager.(*SyncManager).messagesHandler()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:1308 +0x9da

Goroutine 429 (running) created at:
  net/http.(*Server).Serve()
      /usr/local/Cellar/go/1.11.2/libexec/src/net/http/server.go:2851 +0x4c5
  github.com/copernet/copernicus/rpc.(*Server).Start.func2()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:455 +0xcd

Goroutine 70 (running) created at:
  github.com/copernet/copernicus/net/syncmanager.(*SyncManager).Start()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:1544 +0xca
  github.com/copernet/copernicus/net/server.(*Server).cycle()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/net/server/server.go:2056 +0xc3
==================
==================
WARNING: DATA RACE
Read at 0x00c001a15770 by goroutine 429:
  github.com/copernet/copernicus/persist/disk.FlushStateToDisk()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/persist/disk/disk.go:398 +0x995
  github.com/copernet/copernicus/rpc.handleGetTxoutSetInfo()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/rpc/rpcblockchain.go:755 +0x55
  github.com/copernet/copernicus/rpc.(*Server).standardCmdResult()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:217 +0x13b
  github.com/copernet/copernicus/rpc.(*Server).jsonRPCRead()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:378 +0xbdb
  github.com/copernet/copernicus/rpc.(*Server).Start.func1()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:448 +0x212
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/Cellar/go/1.11.2/libexec/src/net/http/server.go:1964 +0x51
  net/http.(*ServeMux).ServeHTTP()
      /usr/local/Cellar/go/1.11.2/libexec/src/net/http/server.go:2361 +0x191
  net/http.serverHandler.ServeHTTP()
      /usr/local/Cellar/go/1.11.2/libexec/src/net/http/server.go:2741 +0xc4
  net/http.(*conn).serve()
      /usr/local/Cellar/go/1.11.2/libexec/src/net/http/server.go:1847 +0x80a
Previous write at 0x00c001a15770 by goroutine 70:
  github.com/copernet/copernicus/persist.(*PersistGlobal).AddDirtyBlockIndex()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/persist/global.go:56 +0x99
  github.com/copernet/copernicus/model/chain.(*Chain).AddToIndexMap()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/model/chain/chain.go:594 +0x409
  github.com/copernet/copernicus/logic/lblock.AcceptBlockHeader()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:311 +0x5d7
  github.com/copernet/copernicus/service.ProcessBlockHeader()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/service/blockservice.go:19 +0x135
  github.com/copernet/copernicus/net/syncmanager.(*SyncManager).handleHeadersMsg()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:946 +0x52d
  github.com/copernet/copernicus/net/syncmanager.(*SyncManager).messagesHandler()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:1308 +0x9da

Goroutine 429 (running) created at:
  net/http.(*Server).Serve()
      /usr/local/Cellar/go/1.11.2/libexec/src/net/http/server.go:2851 +0x4c5
  github.com/copernet/copernicus/rpc.(*Server).Start.func2()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/rpc/rpcserver.go:455 +0xcd

Goroutine 70 (running) created at:
  github.com/copernet/copernicus/net/syncmanager.(*SyncManager).Start()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:1544 +0xca
  github.com/copernet/copernicus/net/server.(*Server).cycle()
      /Users/xiaolong.ran/go/src/github.com/copernet/copernicus/net/server/server.go:2056 +0xc3
==================
fatal error: concurrent map iteration and map write

the exporter of copernicus is a long connect that repeatedly calls the form of RPC, the error:

2018/12/17 15:47:53.300 [E]  EOF
2018/12/17 15:47:53.301 [E]  EOF
2018/12/17 15:47:53.303 [E]  EOF
2018/12/17 15:47:58.307 [E]  EOF
2018/12/17 15:47:58.309 [E]  EOF
2018/12/17 15:47:58.317 [E]  EOF
2018/12/17 15:47:58.317 [E]  EOF
2018/12/17 15:47:58.320 [E]  EOF

This problem can be repeated over and over again.

function @LimitMempoolSize not ready

Describe the bug
Function LimitMempoolSize is not ready and hasn't been invoked yet.

To Reproduce

Expected behavior

Screenshots

Desktop (please complete the following information):

Smartphone (please complete the following information):

Additional context

CheckFee always return 0

Describe the bug
The function should work with some config and should not always return 0.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

unittest in ./check.sh failed, when there is a Copernicus running on localhost

Describe the bug
unittest in ./check.sh failed, when there is a copernicus is running on localhost

To Reproduce
Steps to reproduce the behavior:

  1. start a copernicus instance on localhost with default mainnet parameter
  2. running ./check.sh

Additional context

| => ./check.sh 2>&1 | grep -v "ld: warning: text-based stub file"
++ type -p glide
+ '[' '!' -x /usr/local/bin/glide ']'
++ type -p gometalinter
+ '[' '!' -x /Users/bj1809120051/.go/bin/gometalinter ']'
+ find . -name '*.go' -not -path './vendor/*' -not -path './git/*'
+ xargs gofmt -w
++ glide novendor
+ linter_targets='./crypto/...
./tools/...
./net/...
./util/...
./logic/...
./peer/...
./model/...
./log/...
./service/...
./rpc/...
./conf/...
./persist/...
./errcode/...
.'
++ gometalinter -j 4 --disable-all --enable=gofmt --enable=golint --enable=gosimple --enable=ineffassign --enable=vet --enable=misspell --enable=unconvert '--exclude=should have comment' '--exclude=and that stutters;' --deadline=10m ./crypto/... ./tools/... ./net/... ./util/... ./logic/... ./peer/... ./model/... ./log/... ./service/... ./rpc/... ./conf/... ./persist/... ./errcode/... .
++ grep -v 'ALL_CAPS\|OP_'
++ tee /dev/stderr
+ test -z ''
+ go test -covermode=atomic -coverprofile=coverage.out -race -tags rpctest ./crypto/... ./tools/... ./net/... ./util/... ./logic/... ./peer/... ./model/... ./log/... ./service/... ./rpc/... ./conf/... ./persist/... ./errcode/... .
# github.com/copernet/copernicus/logic/lmerkleroot.test
# github.com/copernet/copernicus/model/txin.test
# github.com/copernet/copernicus/crypto.test
# github.com/copernet/copernicus/net/addrmgr.test
# github.com/copernet/copernicus/model/script.test
# github.com/copernet/copernicus/net/connmgr.test
# github.com/copernet/copernicus/util/cashaddr.test
# github.com/copernet/copernicus/util/bloom.test
# github.com/copernet/copernicus/util/wif.test
# github.com/copernet/copernicus/net/syncmanager.test
# github.com/copernet/copernicus/net/wire.test
# github.com/copernet/copernicus/logic/lblock.test
# github.com/copernet/copernicus/net/server.test
# github.com/copernet/copernicus/logic/lblockindex.test
ok  	github.com/copernet/copernicus/crypto	10.080s	coverage: 77.1% of statements
?   	github.com/copernet/copernicus/tools/blockanalyse	[no test files]
?   	github.com/copernet/copernicus/tools/coperctl	[no test files]
# github.com/copernet/copernicus/logic/lchain.test
# github.com/copernet/copernicus/logic/lmempool.test
# github.com/copernet/copernicus/logic/lmerkleblock.test
# github.com/copernet/copernicus/logic/lscript.test
# github.com/copernet/copernicus/logic/lreindex.test
ok  	github.com/copernet/copernicus/net/addrmgr	20.027s	coverage: 83.5% of statements
ok  	github.com/copernet/copernicus/net/connmgr	10.962s	coverage: 88.0% of statements
ok  	github.com/copernet/copernicus/net/limits	1.337s	coverage: 26.3% of statements
# github.com/copernet/copernicus/logic/lundo.test
# github.com/copernet/copernicus/model/block.test
# github.com/copernet/copernicus/logic/ltx.test
# github.com/copernet/copernicus/model.test
# github.com/copernet/copernicus/model/blockindex.test
ok  	github.com/copernet/copernicus/net/server	17.292s	coverage: 72.7% of statements
?   	github.com/copernet/copernicus/net/socks	[no test files]
ok  	github.com/copernet/copernicus/net/syncmanager	13.897s	coverage: 80.8% of statements
?   	github.com/copernet/copernicus/net/upnp	[no test files]
ok  	github.com/copernet/copernicus/net/wire	10.792s	coverage: 87.8% of statements
ok  	github.com/copernet/copernicus/util	14.845s	coverage: 90.8% of statements
?   	github.com/copernet/copernicus/util/algorithm/mapcontainer	[no test files]
ok  	github.com/copernet/copernicus/util/algorithm/mapcontainer/skiplist	4.297s	coverage: 100.0% of statements
ok  	github.com/copernet/copernicus/util/amount	11.421s	coverage: 95.7% of statements
ok  	github.com/copernet/copernicus/util/base58	1.256s	coverage: 100.0% of statements
ok  	github.com/copernet/copernicus/util/bloom	9.533s	coverage: 86.5% of statements
ok  	github.com/copernet/copernicus/util/cashaddr	8.880s	coverage: 80.7% of statements
ok  	github.com/copernet/copernicus/util/wif	9.861s	coverage: 81.0% of statements
# github.com/copernet/copernicus/model/chain.test
ok  	github.com/copernet/copernicus/logic/lblock	19.555s	coverage: 60.5% of statements
ok  	github.com/copernet/copernicus/logic/lblockindex	10.592s	coverage: 84.5% of statements
# github.com/copernet/copernicus/peer.test
ok  	github.com/copernet/copernicus/logic/lchain	14.419s	coverage: 79.1% of statements
# github.com/copernet/copernicus/model/pow.test
ok  	github.com/copernet/copernicus/logic/lmempool	16.248s	coverage: 79.8% of statements
ok  	github.com/copernet/copernicus/logic/lmerkleblock	1.510s	coverage: 84.2% of statements
ok  	github.com/copernet/copernicus/logic/lmerkleroot	2.211s	coverage: 89.2% of statements
ok  	github.com/copernet/copernicus/logic/lreindex	1.446s	coverage: 85.3% of statements
ok  	github.com/copernet/copernicus/logic/lscript	2.888s	coverage: 84.2% of statements
# github.com/copernet/copernicus/model/tx.test
# github.com/copernet/copernicus/model/txout.test
# github.com/copernet/copernicus/model/undo.test
# github.com/copernet/copernicus/model/utxo.test
# github.com/copernet/copernicus/model/versionbits.test
# github.com/copernet/copernicus/service.test
# github.com/copernet/copernicus/service/mining.test
# github.com/copernet/copernicus/persist.test
# github.com/copernet/copernicus/persist/blkdb.test
# github.com/copernet/copernicus/persist/disk.test
# github.com/copernet/copernicus/model/mempool.test
ok  	github.com/copernet/copernicus/logic/ltx	35.545s	coverage: 86.1% of statements
ok  	github.com/copernet/copernicus/logic/lundo	8.759s	coverage: 85.1% of statements
?   	github.com/copernet/copernicus/logic/lwallet	[no test files]
2018/12/06 21:42:19.399 [D]  AddToIndexMap:000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f index height:0
2018/12/06 21:42:19.400 [D]  &{GoVersion:1.9.2 Version:1.0.0 BuildDate:20180428 DataDir:/Users/bj1809120051/Library/Application Support/Bitcoincash Reindex:false Excessiveblocksize:32000000 RPC:{RPCListeners:[127.0.0.1:18334] RPCUser:copernicus RPCPass:doXT3DXgAQCNU0Li0pujQ6zR3Y RPCLimitUser: RPCLimitPass: RPCCert:/Users/bj1809120051/Library/Application Support/Bitcoincash/rpc.cert RPCKey:/Users/bj1809120051/Library/Application Support/Bitcoincash/rpc.key RPCMaxClients:1000 RPCMaxWebsockets:0 RPCMaxConcurrentReqs:0 RPCQuirks:false} Log:{Level:error Module:[mempool utxo bench service] FileName:copernicus} Mempool:{MinFeeRate:0 LimitAncestorCount:50000 LimitAncestorSize:5000 LimitDescendantCount:50000 LimitDescendantSize:5000 MaxPoolSize:300000000 MaxPoolExpiry:336 CheckFrequency:4294967296} P2PNet:{ListenAddrs:[127.0.0.1:18333] MaxPeers:128 TargetOutbound:64 ConnectPeersOnStart:[] DisableBanning:true BanThreshold:100 TestNet:false RegTest:false SimNet:false DisableListen:false BlocksOnly:false BanDuration:86400 Proxy: UserAgentComments:[] DisableDNSSeed:false DisableRPC:false DisableTLS:false Whitelists:[] NoOnion:true Upnp:false ExternalIPs:[] MaxTimeAdjustment:4200} AddrMgr:{SimNet:false ConnectPeers:[]} Protocol:{NoPeerBloomFilters:true DisableCheckpoints:true} Script:{AcceptDataCarrier:true MaxDatacarrierBytes:223 IsBareMultiSigStd:true PromiscuousMempoolFlags: Par:32} TxOut:{DustRelayFee:83} Chain:{AssumeValid: UtxoHashStartHeight:-1 UtxoHashEndHeight:-1} Mining:{BlockMinTxFee:100 BlockMaxSize:2000000 Strategy:ancestorfeerate} PProf:{IP:localhost Port:6060} BlockIndex:{CheckBlockIndex:false} Wallet:{Enable:false Broadcast:false SpendZeroConfChange:true}}
2018/12/06 21:42:19.402 [E]  Can't listen on 127.0.0.1:18333: listen tcp4 127.0.0.1:18333: bind: address already in use
panic: server init fail

goroutine 1 [running]:
github.com/copernet/copernicus/peer_test.initEnv()
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/peer/peer_test.go:265 +0xb71
github.com/copernet/copernicus/peer_test.TestMain(0xc000125300)
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/peer/peer_test.go:52 +0x89
main.main()
	_testmain.go:116 +0x32f
FAIL	github.com/copernet/copernicus/peer	10.536s
ok  	github.com/copernet/copernicus/model	9.843s	coverage: 87.2% of statements
ok  	github.com/copernet/copernicus/model/block	8.745s	coverage: 84.8% of statements
ok  	github.com/copernet/copernicus/model/blockindex	56.925s	coverage: 77.9% of statements
ok  	github.com/copernet/copernicus/model/chain	11.324s	coverage: 79.7% of statements
ok  	github.com/copernet/copernicus/model/consensus	7.979s	coverage: 100.0% of statements
ok  	github.com/copernet/copernicus/model/mempool	1.546s	coverage: 83.6% of statements
ok  	github.com/copernet/copernicus/model/opcodes	1.297s	coverage: 90.3% of statements
ok  	github.com/copernet/copernicus/model/outpoint	11.491s	coverage: 100.0% of statements
ok  	github.com/copernet/copernicus/model/pow	11.744s	coverage: 81.5% of statements
ok  	github.com/copernet/copernicus/model/script	1.597s	coverage: 87.4% of statements
ok  	github.com/copernet/copernicus/model/tx	13.628s	coverage: 78.2% of statements
ok  	github.com/copernet/copernicus/model/txin	9.057s	coverage: 100.0% of statements
ok  	github.com/copernet/copernicus/model/txout	3.709s	coverage: 86.7% of statements
ok  	github.com/copernet/copernicus/model/undo	10.272s	coverage: 96.8% of statements
ok  	github.com/copernet/copernicus/model/utxo	2.500s	coverage: 80.2% of statements
ok  	github.com/copernet/copernicus/model/versionbits	1.510s	coverage: 6.7% of statements
?   	github.com/copernet/copernicus/model/wallet	[no test files]
ok  	github.com/copernet/copernicus/log	1.578s	coverage: 97.6% of statements
ok  	github.com/copernet/copernicus/service	12.905s	coverage: 81.7% of statements
ok  	github.com/copernet/copernicus/service/mining	13.191s	coverage: 85.1% of statements
?   	github.com/copernet/copernicus/rpc	[no test files]
ok  	github.com/copernet/copernicus/rpc/btcjson	1.371s	coverage: 99.3% of statements
ok  	github.com/copernet/copernicus/conf	11.770s	coverage: 78.6% of statements
ok  	github.com/copernet/copernicus/persist	1.459s	coverage: 91.7% of statements
ok  	github.com/copernet/copernicus/persist/blkdb	1.464s	coverage: 78.0% of statements
ok  	github.com/copernet/copernicus/persist/db	12.185s	coverage: 84.3% of statements
ok  	github.com/copernet/copernicus/persist/disk	1.681s	coverage: 62.1% of statements
ok  	github.com/copernet/copernicus/errcode	1.514s	coverage: 82.6% of statements
?   	github.com/copernet/copernicus	[no test files]

Send GetHeaders msg once to every peer when pindexBestHeader is 24h near to now

Describe the solution you'd like
modify the scanToFetchHeaderBlocks func, add a method in it to check whether the condition is ok to send this GetHeaders request

Additional context
Without this feature the current code is ok to run.
But with this feature, the code will be much more robust.
Because this will let us to know all available headers, thus we can find the most pow work header to sync to.

UTXO check

Compare whether the changes to UTXO are consistent after each block has been processed

copernicus not quit gracefully when recevied the `stop' rpc from copernet/walle functional test.

-68-> stop []
<-68- [0.000525] "Copernicus server stopping"
2018-09-28 10:56:18.080000 TestFramework.node1 (DEBUG): Stopping node
-69-> stop []
<-69- [0.000473] "Copernicus server stopping"
2018-09-28 10:56:18.081000 TestFramework.node2 (DEBUG): Stopping node
-70-> stop []
<-70- [0.000623] "Copernicus server stopping"
Traceback (most recent call last):
File "abc-rpc.py", line 58, in
ABC_RPC_Test().main()
File "/Users/bj1809120051/lab/walle/test_framework/test_framework.py", line 154, in main
self.stop_nodes()
File "/Users/bj1809120051/lab/walle/test_framework/test_framework.py", line 295, in stop_nodes
node.wait_until_stopped()
File "/Users/bj1809120051/lab/walle/test_framework/test_node.py", line 165, in wait_until_stopped
wait_until(self.is_node_stopped, timeout=timeout)
File "/Users/bj1809120051/lab/walle/test_framework/util.py", line 252, in wait_until
assert_greater_than(timeout, time.time())
File "/Users/bj1809120051/lab/walle/test_framework/util.py", line 48, in assert_greater_than
raise AssertionError("%s <= %s" % (str(thing1), str(thing2)))
AssertionError: 1538132238.082943 <= 1538132238.118457

BanDuration has no default value and is not configured

Describe the bug
conf.Cfg.P2PNet.BanDuration has no default value and is not configured

To Reproduce
review the code in conf.go

Expected behavior
set a default value

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: linux

Unable to checkout golang.org/x/crypto

[WARN] Unable to checkout golang.org/x/crypto
[ERROR] Update failed for golang.org/x/crypto: Cannot detect VCS
[ERROR] Failed to do initial checkout of config: Cannot detect VCS

Cash address misses prefix

util/cashaddr/cashAddr.go https://github.com/copernet/copernicus/blob/master/util/cashaddr/cashAddr.go
function 'Encode' (Line260 - 270)

func Encode(prefix string, payload Data) string {
	checksum := CreateChecksum(prefix, payload)
	combined := Cat(payload, checksum)
	ret := ""
	for _, c := range combined {
		ret += string(CHARSET[c])
	}
	return ret
}

Encoded address has no prefix. Is it a bug?

Refer to bitcoin-abc https://github.com/Bitcoin-ABC/bitcoin-abc/blob/master/src/cashaddr.cpp
src/cashaddr.cpp function 'Encode' (Line205 - 215)

std::string Encode(const std::string &prefix, const data &payload) {
    data checksum = CreateChecksum(prefix, payload);
    data combined = Cat(payload, checksum);
    std::string ret = prefix + ':';
    ret.reserve(ret.size() + combined.size());
    for (uint8_t c : combined) {
        ret += CHARSET[c];
    }
    return ret;
}

Self address broadcast function

Describe the solution you'd like
We should do self addr broadcast via ADDR msg in two situations:
1, OnVersion, when recv version from peer
2, Periodically, refresh to broadcast.

Additional context
For detail info, ref abc.

Bug of startSync() in syncmanager, when the node has only one peer

Describe the bug
When the node has only one peer, such as you set conf to let the node to connect only a local peer,
a bug of startSync() may occur, that you select syncPeer fail, and then you will never get syncPeer.

To Reproduce
Steps to reproduce the behavior:

  1. select only a local peer
  2. start copernicus
  3. the bug may occur probably

Expected behavior
Node should select this only peer as syncPeer to download headers and blocks.

Detailed RevertToInv check function

Is your feature request related to a problem? Please describe.
Upon several situation, we should revert to inv, instead of headers, when relay blocks.
If not, we may misbehave.

Describe the solution you'd like
1, Check whether the blockindex is in active chain.
2, Check whether the blocks to relay connect one by one.
3, Check whether the peer has the prev of first header we are to relay.

send inv msg to fetch missing tx

Describe the bug
send inv msg to fetch missing tx

To Reproduce
function fetchMissingTx in syncmanager.go

Expected behavior
send getdata msg

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS:linux

Crashed when sync block on testnet.

Describe the bug
A clear and concise description of what the bug is.

Processed block: 000000000004a714d5895f0810b2076d8f19fcca137d0f3b996661c2bdd64c9c, Chain height: 240571, tipHash: 000000000004a714d5895f0810b2076d8f19fcca137d0f3b996661c2bdd64c9c, coinsTip hash: 000000000004a714d5895f0810b2076d8f19fcca137d0f3b996661c2bdd64c9c currenttime:2018-11-26 17:51:09.526121 +0800 CST m=+22985.273894318
Processed block: 00000000000b4388fed5012d086c7d37c74b633455b313ca2770b37fe807a66e, Chain height: 240572, tipHash: 00000000000b4388fed5012d086c7d37c74b633455b313ca2770b37fe807a66e, coinsTip hash: 00000000000b4388fed5012d086c7d37c74b633455b313ca2770b37fe807a66e currenttime:2018-11-26 17:51:09.614594 +0800 CST m=+22985.362367530
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x465089a]

goroutine 44131 [running]:
github.com/copernet/copernicus/model/chain.(*Chain).TipHeight(0xc0000b60a0, 0xc000000000)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/model/chain/chain.go:160 +0x5a
github.com/copernet/copernicus/net/server.(*serverPeer).newestBlock(0xc0049b8bd0, 0x0, 0x0, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:332 +0x72
github.com/copernet/copernicus/net/server.(*serverPeer).newestBlock-fm(0x0, 0x0, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/net/server/server.go:1871 +0x68
github.com/copernet/copernicus/peer.(*Peer).localVersionMsg(0xc0039da900, 0x0, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:839 +0x19c
github.com/copernet/copernicus/peer.(*Peer).writeLocalVersionMsg(0xc0039da900, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:2015 +0x51
github.com/copernet/copernicus/peer.(*Peer).negotiateOutboundProtocol(0xc0039da900, 0x0, 0x0)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:2038 +0x37
github.com/copernet/copernicus/peer.(*Peer).start.func1(0xc0039da900, 0xc00024de60)
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:1936 +0x18e
created by github.com/copernet/copernicus/peer.(*Peer).start
	/Users/zhangyong/go/src/github.com/copernet/copernicus/peer/peer.go:1931 +0x12c

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

too much log output related to fetchHeaderBlocks

Describe the bug
too many log related to fetchHeaderBlocks

To Reproduce
Steps to reproduce the behavior:

  1. running latest version, e.g.: 3a07937

Expected behavior
should not have such overwhelming logs which will take too much disk space and slow down the ELK logging system.

Additional context


2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1712) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1712)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(2486) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2486)
2018/12/10 10:20:31.391 [I] [syncmanager.go:766]  peer(2494) ChainWork less than minChainWork, do not use this peer
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1007) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1007)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(2089) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2089)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(2152) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2152)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(2214) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2214)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1004) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1004)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1619) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1619)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(606) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(606)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(701) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(701)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1436) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1436)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1299) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1299)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1678) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1678)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1944) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1944)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(757) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(757)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(201) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(201)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(947) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(947)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1488) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1488)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(2457) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2457)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1430) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1430)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1287) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1287)
2018/12/10 10:20:31.391 [I] [syncmanager.go:685]  peer(1940) best known block nil, forgive temporary
2018/12/10 10:20:31.391 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.391 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1940)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(2096) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2096)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(2119) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2119)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(1626) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1626)
2018/12/10 10:20:31.392 [D] [peer.go:1211]  write summary getheaders (locator 00000000000002aaa2874bb02228fc96f6bb0654d1d4fd29ce4ecb9dcf36cf7c, stop 0000000000000000000000000000000000000000000000000000000000000000) to 116.125.120.98:18333 (outbound)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(2177) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2177)
2018/12/10 10:20:31.392 [I] [syncmanager.go:766]  peer(22) ChainWork less than minChainWork, do not use this peer
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(771) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(771)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(855) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(855)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(1306) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1306)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(1405) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1405)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(1474) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1474)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(2491) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2491)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(2961) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2961)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(59) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(59)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(459) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(459)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(2580) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [peer.go:1211]  write summary getheaders (locator 00000000000002aaa2874bb02228fc96f6bb0654d1d4fd29ce4ecb9dcf36cf7c, stop 0000000000000000000000000000000000000000000000000000000000000000) to 142.93.111.194:18333 (outbound)
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2580)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(2602) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2602)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(2009) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2009)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(109) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(109)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(113) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(113)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(399) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(399)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(1807) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1807)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(2357) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.392 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2357)
2018/12/10 10:20:31.392 [I] [syncmanager.go:685]  peer(1209) best known block nil, forgive temporary
2018/12/10 10:20:31.392 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1209)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2443) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2443)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(178) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(178)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(214) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(214)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(574) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(574)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2502) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2502)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2555) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2555)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2829) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2829)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2702) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2702)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2799) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2799)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(760) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(760)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(1354) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1354)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(1416) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1416)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2226) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2226)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(573) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(573)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2037) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2037)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2317) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2317)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2098) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2098)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2105) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2105)
2018/12/10 10:20:31.393 [I] [syncmanager.go:766]  peer(27) ChainWork less than minChainWork, do not use this peer
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(203) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(203)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(1058) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1058)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(1060) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1060)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2644) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2644)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2540) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2540)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(140) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(140)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(1366) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1366)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(1490) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1490)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(1858) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1858)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2155) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2155)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2294) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2294)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2323) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2323)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(2781) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2781)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(261) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(261)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(1937) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1937)
2018/12/10 10:20:31.393 [I] [syncmanager.go:685]  peer(1195) best known block nil, forgive temporary
2018/12/10 10:20:31.393 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.393 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1195)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(476) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(476)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(487) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(487)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(3007) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(3007)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2845) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2845)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2854) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2854)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2404) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2404)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2878) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2878)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(352) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(352)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(1497) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1497)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(560) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(560)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(1292) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1292)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2230) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2230)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2630) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2630)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(100) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(100)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(491) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(491)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(818) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(818)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2390) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2390)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(154) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(154)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(977) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(977)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(1645) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1645)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2275) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2275)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2733) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2733)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(655) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(655)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(1227) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1227)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(1451) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1451)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(1744) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1744)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2699) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2699)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(60) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(60)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(880) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(880)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(543) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(543)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(2350) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.394 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2350)
2018/12/10 10:20:31.394 [I] [syncmanager.go:685]  peer(314) best known block nil, forgive temporary
2018/12/10 10:20:31.394 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(314)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(1709) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1709)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(1575) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1575)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(2909) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2909)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(2982) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2982)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(2789) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2789)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(368) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(368)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(383) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(383)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(961) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(961)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(1022) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1022)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(1898) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1898)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(2776) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2776)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(1862) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1862)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(2760) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2760)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(170) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(170)
2018/12/10 10:20:31.395 [I] [syncmanager.go:685]  peer(2134) best known block nil, forgive temporary
2018/12/10 10:20:31.395 [D] [chain.go:410]  GetLocator from hash : 00000000000000389bdc7efd5184063ca7b75319f0a09ee1a1e1583571a08f34, height : 1273262 .
2018/12/10 10:20:31.395 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2134)

:~/.bitcoincash/testnet/logs# head copernicus.2018-12-10.001.log
2018/12/09 23:09:05.389 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(1354)
2018/12/09 23:09:05.389 [I] [syncmanager.go:685]  peer(573) best known block nil, forgive temporary
2018/12/09 23:09:05.389 [D] [chain.go:410]  GetLocator from hash : 000000000000072e6ba253c6ceed49ddf6e89cfa83c67c7c0aa33deaf9da45f8, height : 1273182 .
2018/12/09 23:09:05.389 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(573)
2018/12/09 23:09:05.389 [I] [syncmanager.go:685]  peer(2037) best known block nil, forgive temporary
2018/12/09 23:09:05.389 [D] [chain.go:410]  GetLocator from hash : 000000000000072e6ba253c6ceed49ddf6e89cfa83c67c7c0aa33deaf9da45f8, height : 1273182 .
2018/12/09 23:09:05.389 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(2037)
2018/12/09 23:09:05.389 [I] [syncmanager.go:685]  peer(203) best known block nil, forgive temporary
2018/12/09 23:09:05.389 [D] [chain.go:410]  GetLocator from hash : 000000000000072e6ba253c6ceed49ddf6e89cfa83c67c7c0aa33deaf9da45f8, height : 1273182 .
2018/12/09 23:09:05.389 [D] [syncmanager.go:761]  fetchHeaderBlocks can not find block hashes to fetch from peer(203)
:~/.bitcoincash/testnet/logs# date
Mon Dec 10 10:27:48 CST 2018
:~/.bitcoincash/testnet/logs# du -sh copernicus.2018-12-10*
199M	copernicus.2018-12-10.001.log
257M	copernicus.2018-12-10.002.log
257M	copernicus.2018-12-10.003.log
257M	copernicus.2018-12-10.004.log
257M	copernicus.2018-12-10.005.log
257M	copernicus.2018-12-10.006.log
257M	copernicus.2018-12-10.007.log
257M	copernicus.2018-12-10.008.log
257M	copernicus.2018-12-10.009.log
257M	copernicus.2018-12-10.010.log
257M	copernicus.2018-12-10.011.log
257M	copernicus.2018-12-10.012.log
257M	copernicus.2018-12-10.013.log

data race during unittest /copernicus/net/server.TestOnFilterAdd

Describe the bug
data race during unit test

To Reproduce
execute ./check.sh

Expected behavior
no data race breaks the unit test

Additional context


ok  	github.com/[secure]/copernicus/net/limits	1.024s	coverage: 26.3% of statements
Current data dir: /tmp/server774522591/regtest 
==================
WARNING: DATA RACE
Write at 0x00c4200bc730 by goroutine 29:
  github.com/[secure]/copernicus/net/server.TestOnFilterAdd()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:886 +0x1ba
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:777 +0x16d
Previous read at 0x00c4200bc730 by goroutine 23:
  github.com/[secure]/copernicus/net/server.(*Server).upnpUpdateThread()
      /go/src/github.com/[secure]/copernicus/net/server/server.go:2537 +0x4c0
Goroutine 29 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:824 +0x564
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1063 +0xa4
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:777 +0x16d
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1061 +0x4e1
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:978 +0x2cd
  github.com/[secure]/copernicus/net/server.TestMain()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:165 +0x191
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_util.go:52 +0x2e6
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:130 +0xa4c
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:657 +0x239
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:476 +0xd9
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:125 +0xa28
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:197 +0x186
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/session.go:113 +0xc3
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/session.go:121 +0x1860
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:183 +0xc8
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.OpenFile()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:219 +0x9f
  github.com/[secure]/copernicus/persist/db.NewDBWrapper()
      /go/src/github.com/[secure]/copernicus/persist/db/db.go:158 +0x43c
  github.com/[secure]/copernicus/persist/blkdb.newBlockTreeDB()
      /go/src/github.com/[secure]/copernicus/persist/blkdb/blkdb.go:42 +0x59
  github.com/[secure]/copernicus/persist/blkdb.InitBlockTreeDB()
      /go/src/github.com/[secure]/copernicus/persist/blkdb/blkdb.go:28 +0x4d
  github.com/[secure]/copernicus/net/server.appInitMain()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:108 +0x87e
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_util.go:52 +0x2e6
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:130 +0xa4c
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:657 +0x239
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:476 +0xd9
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:125 +0xa28
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:197 +0x186
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/session.go:113 +0xc3
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/session.go:121 +0x1860
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:183 +0xc8
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.OpenFile()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:219 +0x9f
  github.com/[secure]/copernicus/persist/db.NewDBWrapper()
      /go/src/github.com/[secure]/copernicus/persist/db/db.go:158 +0x43c
  github.com/[secure]/copernicus/model/utxo.newCoinsDB()
      /go/src/github.com/[secure]/copernicus/model/utxo/coindb.go:122 +0x59
  github.com/[secure]/copernicus/model/utxo.InitUtxoLruTip()
      /go/src/github.com/[secure]/copernicus/model/utxo/lrucache.go:25 +0x4d
  github.com/[secure]/copernicus/net/server.appInitMain()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:97 +0x71e
  github.com/[secure]/copernicus/net/server.makeTestServer()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:142 +0x14c
  github.com/[secure]/copernicus/net/server.TestMain()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:157 +0x44
  main.main()
Goroutine 23 (running) created at:
  github.com/[secure]/copernicus/net/server.(*Server).Start()
      /go/src/github.com/[secure]/copernicus/net/server/server.go:2393 +0x369
  github.com/[secure]/copernicus/net/server.TestMain()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:164 +0x180
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_util.go:52 +0x2e6
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:130 +0xa4c
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:657 +0x239
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:476 +0xd9
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:125 +0xa28
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:197 +0x186
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/session.go:113 +0xc3
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/session.go:121 +0x1860
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:183 +0xc8
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.OpenFile()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:219 +0x9f
  github.com/[secure]/copernicus/persist/db.NewDBWrapper()
      /go/src/github.com/[secure]/copernicus/persist/db/db.go:158 +0x43c
  github.com/[secure]/copernicus/persist/blkdb.newBlockTreeDB()
      /go/src/github.com/[secure]/copernicus/persist/blkdb/blkdb.go:42 +0x59
  github.com/[secure]/copernicus/persist/blkdb.InitBlockTreeDB()
      /go/src/github.com/[secure]/copernicus/persist/blkdb/blkdb.go:28 +0x4d
  github.com/[secure]/copernicus/net/server.appInitMain()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:108 +0x87e
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db_util.go:52 +0x2e6
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:130 +0xa4c
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:657 +0x239
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:476 +0xd9
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.openDB()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:125 +0xa28
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:197 +0x186
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x345
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xf5
  fmt.Sscanf()
      /usr/local/go/src/fmt/scan.go:114 +0xbb
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:643 +0x16c
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:458 +0x2b0
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/session.go:113 +0xc3
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/session.go:121 +0x1860
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.Open()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:183 +0xc8
  github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb.OpenFile()
      /go/src/github.com/[secure]/copernicus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:219 +0x9f
  github.com/[secure]/copernicus/persist/db.NewDBWrapper()
      /go/src/github.com/[secure]/copernicus/persist/db/db.go:158 +0x43c
  github.com/[secure]/copernicus/model/utxo.newCoinsDB()
      /go/src/github.com/[secure]/copernicus/model/utxo/coindb.go:122 +0x59
  github.com/[secure]/copernicus/model/utxo.InitUtxoLruTip()
      /go/src/github.com/[secure]/copernicus/model/utxo/lrucache.go:25 +0x4d
  github.com/[secure]/copernicus/net/server.appInitMain()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:97 +0x71e
  github.com/[secure]/copernicus/net/server.makeTestServer()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:142 +0x14c
  github.com/[secure]/copernicus/net/server.TestMain()
      /go/src/github.com/[secure]/copernicus/net/server/server_test.go:157 +0x44
  main.main()
      _testmain.go:234 +0x320
==================
--- FAIL: TestOnFilterAdd (0.00s)
	testing.go:730: race detected during execution of test
FAIL
coverage: 72.3% of statements

Rpc respond 503 Too busy

Describe the bug
coperctl --notls -s xxx.xxx.xxx.xxx:xxx help
503 Too busy

but there is just one connection to copernicus, and there is none log with that problem

misbehave re-checking during p2p msg sending & recving

Describe the bug
After long time running, it seems we may be banned by some peers.

To Reproduce
Difficult to reproduce, should run for a long time, and maybe occur.

Expected behavior
Should work normally even after long time running.

Additional context
I will do a test, let copernicus only connect to a local abc peer, modify abc's log to print out the misbehaving related logs, to check if we have ever misbehaved.

Unit test 'TestAcceptBlockHeader' failed

Describe the bug
--- FAIL: TestAcceptBlockHeader (0.00s)
lblock_test.go:185: TestAcceptBlockHeader test 6 check checkpoint height failed

To Reproduce
run 'lblock' unit test

Expected behavior
unit test pass

restore block state, after write block content to disk failed.

Describe the bug
when write block content to disk failed, the block state should be restored.
also, should not panic the P2P block handling thread.

To Reproduce
run tests in walle on ubuntu virtual machine, occasionally will see file write error

j@j:~/lab/walleubuntu$ pipenv run python test_runner.py

Expected behavior
gracefully handle block file write failure.

Screenshots

j@j:~/lab/walleubuntu$ pipenv run python test_runner.py
Temporary test directory at /tmp/bitcoin_test_runner_20181028_011425






2018/10/28 01:15:31 http: panic serving 127.0.0.1:59280: AcceptBlockHeader WriteBlockTo Disk errmodule: , global errcode: 2001,  desc:
goroutine 852 [running]:
net/http.(*conn).serve.func1(0xc42021e1e0)
	/usr/lib/go-1.10/src/net/http/server.go:1726 +0x11b
panic(0xc64320, 0xc420056660)
	/usr/lib/go-1.10/src/runtime/panic.go:502 +0x24a
github.com/copernet/copernicus/logic/lblock.AcceptBlock(0xc420172a50, 0xd9b501, 0x0, 0xc4213311fe, 0xc4206d7f00, 0x0, 0xde9400, 0xc4201e4040)
	/home/j/.go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:265 +0x8b7
github.com/copernet/copernicus/service.ProcessNewBlock(0xc420172a50, 0xc421331201, 0xc4213311fe, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/service/blockservice.go:75 +0x239
github.com/copernet/copernicus/rpc.generateBlocks(0xc4200b1540, 0x1, 0xf423e, 0x0, 0x0, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/mining.go:590 +0x3f2
github.com/copernet/copernicus/rpc.handleGenerate(0xc420288180, 0xc3f4a0, 0xc420092880, 0xc420df4c00, 0x0, 0x0, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/mining.go:543 +0x1dc
github.com/copernet/copernicus/rpc.(*Server).standardCmdResult(0xc420288180, 0xc4200b1500, 0xc420df4c00, 0x0, 0x0, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:215 +0x114
github.com/copernet/copernicus/rpc.(*Server).jsonRPCRead(0xc420288180, 0xded480, 0xc42007e540, 0xc420dba700, 0x1)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:375 +0x11c5
github.com/copernet/copernicus/rpc.(*Server).Start.func1(0xded480, 0xc42007e540, 0xc420dba700)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:444 +0x269
net/http.HandlerFunc.ServeHTTP(0xc4201f4560, 0xded480, 0xc42007e540, 0xc420dba700)
	/usr/lib/go-1.10/src/net/http/server.go:1947 +0x44
net/http.(*ServeMux).ServeHTTP(0xc4201141e0, 0xded480, 0xc42007e540, 0xc420dba700)
	/usr/lib/go-1.10/src/net/http/server.go:2340 +0x1a3
net/http.serverHandler.ServeHTTP(0xc42009ea90, 0xded480, 0xc42007e540, 0xc420dba700)
	/usr/lib/go-1.10/src/net/http/server.go:2697 +0x1e4
net/http.(*conn).serve(0xc42021e1e0, 0xdeda40, 0xc4200b12c0)
	/usr/lib/go-1.10/src/net/http/server.go:1830 +0x11f2
created by net/http.(*Server).Serve
	/usr/lib/go-1.10/src/net/http/server.go:2798 +0x6e4
2018/10/28 01:15:31 http: panic serving 127.0.0.1:59282: AcceptBlockHeader WriteBlockTo Disk errmodule: , global errcode: 2001,  desc:
goroutine 790 [running]:
net/http.(*conn).serve.func1(0xc42021e0a0)
	/usr/lib/go-1.10/src/net/http/server.go:1726 +0x11b
panic(0xc64320, 0xc4202105d0)
	/usr/lib/go-1.10/src/runtime/panic.go:502 +0x24a
github.com/copernet/copernicus/logic/lblock.AcceptBlock(0xc42136c000, 0xd9b501, 0x0, 0xc4212eb1fe, 0xc4206d7f00, 0x0, 0xde9400, 0xc4200b0f80)
	/home/j/.go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:265 +0x8b7
github.com/copernet/copernicus/service.ProcessNewBlock(0xc42136c000, 0xc4212eb201, 0xc4212eb1fe, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/service/blockservice.go:75 +0x239
github.com/copernet/copernicus/rpc.generateBlocks(0xc4200b1040, 0x1, 0xf423e, 0x0, 0x0, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/mining.go:590 +0x3f2
github.com/copernet/copernicus/rpc.handleGenerate(0xc420288180, 0xc3f4a0, 0xc420210330, 0xc420242480, 0x0, 0x0, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/mining.go:543 +0x1dc
github.com/copernet/copernicus/rpc.(*Server).standardCmdResult(0xc420288180, 0xc4200b0fc0, 0xc420242480, 0x0, 0x0, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:215 +0x114
github.com/copernet/copernicus/rpc.(*Server).jsonRPCRead(0xc420288180, 0xded480, 0xc42007e380, 0xc420e8cb00, 0x1)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:375 +0x11c5
github.com/copernet/copernicus/rpc.(*Server).Start.func1(0xded480, 0xc42007e380, 0xc420e8cb00)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:444 +0x269
net/http.HandlerFunc.ServeHTTP(0xc4201f4560, 0xded480, 0xc42007e380, 0xc420e8cb00)
	/usr/lib/go-1.10/src/net/http/server.go:1947 +0x44
net/http.(*ServeMux).ServeHTTP(0xc4201141e0, 0xded480, 0xc42007e380, 0xc420e8cb00)
	/usr/lib/go-1.10/src/net/http/server.go:2340 +0x1a3
net/http.serverHandler.ServeHTTP(0xc42009ea90, 0xded480, 0xc42007e380, 0xc420e8cb00)
	/usr/lib/go-1.10/src/net/http/server.go:2697 +0x1e4
net/http.(*conn).serve(0xc42021e0a0, 0xdeda40, 0xc4200b0cc0)
	/usr/lib/go-1.10/src/net/http/server.go:1830 +0x11f2
created by net/http.(*Server).Serve
	/usr/lib/go-1.10/src/net/http/server.go:2798 +0x6e4

Desktop (please complete the following information):

  • OS: Linux j 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Additional context
mozilla rr record and debugging logs:



Processed block: 6bf42ca46fbca3f6bc30073340cd706bd223f2151c578efe7eb6d49824e1ea7c, Chain height: 99, tipHash: 6bf42ca46fbca3f6bc30073340cd706bd223f2151c578efe7eb6d49824e1ea7c, coinsTip hash: 6bf42ca46fbca3f6bc30073340cd706bd223f2151c578efe7eb6d49824e1ea7c currenttime:2018-10-27 22:54:39.354403075 +0800 CST m=+53.638146471
Processed block: 53fe66fad35bb02beedb855f3382811588e4465057d63a49a8b895b237a64996, Chain height: 100, tipHash: 53fe66fad35bb02beedb855f3382811588e4465057d63a49a8b895b237a64996, coinsTip hash: 53fe66fad35bb02beedb855f3382811588e4465057d63a49a8b895b237a64996 currenttime:2018-10-27 22:54:39.380722574 +0800 CST m=+53.664466003
2018/10/27 22:54:51 http: panic serving 127.0.0.1:46518: AcceptBlockHeader WriteBlockTo Disk errmodule: , global errcode: 2001,  desc:
goroutine 790 [running]:
net/http.(*conn).serve.func1(0xc4200ac500)
	/usr/lib/go-1.10/src/net/http/server.go:1726 +0x11b
panic(0xc64460, 0xc4201bc640)
	/usr/lib/go-1.10/src/runtime/panic.go:502 +0x24a
github.com/copernet/copernicus/logic/lblock.AcceptBlock(0xc420138fd0, 0xd9b701, 0x0, 0xc4200b71f6, 0xc420dae600, 0x0, 0xde95a0, 0xc4201af500)
	/home/j/.go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:265 +0x8b7
github.com/copernet/copernicus/service.ProcessNewBlock(0xc420138fd0, 0xc4200b7201, 0xc4200b71f6, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/service/blockservice.go:75 +0x239
github.com/copernet/copernicus/rpc.generateBlocks(0xc4201af380, 0x1, 0xf423e, 0x0, 0x0, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/mining.go:593 +0x448
github.com/copernet/copernicus/rpc.handleGenerate(0xc4201ac000, 0xc3f5e0, 0xc4201bc4d0, 0xc420074f60, 0x0, 0x0, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/mining.go:543 +0x1dc
github.com/copernet/copernicus/rpc.(*Server).standardCmdResult(0xc4201ac000, 0xc4201af340, 0xc420074f60, 0x0, 0x0, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:215 +0x114
github.com/copernet/copernicus/rpc.(*Server).jsonRPCRead(0xc4201ac000, 0xded620, 0xc420252540, 0xc420dae300, 0x1)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:375 +0x11c5
github.com/copernet/copernicus/rpc.(*Server).Start.func1(0xded620, 0xc420252540, 0xc420dae300)
	/home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:444 +0x269
net/http.HandlerFunc.ServeHTTP(0xc420056760, 0xded620, 0xc420252540, 0xc420dae300)
	/usr/lib/go-1.10/src/net/http/server.go:1947 +0x44
net/http.(*ServeMux).ServeHTTP(0xc4201e0060, 0xded620, 0xc420252540, 0xc420dae300)
	/usr/lib/go-1.10/src/net/http/server.go:2340 +0x1a3
net/http.serverHandler.ServeHTTP(0xc420085860, 0xded620, 0xc420252540, 0xc420dae300)
	/usr/lib/go-1.10/src/net/http/server.go:2697 +0x1e4
net/http.(*conn).serve(0xc4200ac500, 0xdedbe0, 0xc4201af100)
	/usr/lib/go-1.10/src/net/http/server.go:1830 +0x11f2
created by net/http.(*Server).Serve
	/usr/lib/go-1.10/src/net/http/server.go:2798 +0x6e4
Processed block: 1a9e45d318ca5ad126c78ee389a8375bac5d5c83f6e1a162488f2821ae46304d, Chain height: 126, tipHash: 1a9e45d318ca5ad126c78ee389a8375bac5d5c83f6e1a162488f2821ae46304d, coinsTip hash: 1a9e45d318ca5ad126c78ee389a8375bac5d5c83f6e1a162488f2821ae46304d currenttime:2018-10-27 22:55:07.131758621 +0800 CST m=+81.415502016
Processed block: 6c66f64b5b0e8dcbe6554f256898a12539bedd1a416e1bf0ccaccc0c4d6cf173, Chain height: 127, tipHash: 6c66f64b5b0e8dcbe6554f256898a12539bedd1a416e1bf0ccaccc0c4d6cf173, coinsTip hash: 6c66f64b5b0e8dcbe6554f256898a12539bedd1a416e1bf0ccaccc0c4d6cf173 currenttime:2018-10-27 22:55:07.162631128 +0800 CST m=+81.446374430
Processed block: 0d5cd5be2eb97d8743c34bcbeb01a42fe2dd75d8453874b02549b61307841c7d, Chain height: 128, tipHash: 0d5cd5be2eb97d8743c34bcbeb01a42fe2dd75d8453874b02549b61307841c7d, coinsTip hash: 0d5cd5be2eb97d8743c34bcbeb01a42fe2dd75d8453874b02549b61307841c7d currenttime:2018-10-27 22:55:07.186380017 +0800 CST m=+81.470123396
Processed block: 49f763c448aca7e0bc4494c54e9e40ffe8157526ba6c0a6f2b98a54e906d0d32, Chain height: 129, tipHash: 49f763c448aca7e0bc4494c54e9e40ffe8157526ba6c0a6f2b98a54e906d0d32, coinsTip hash: 49f763c448aca7e0bc4494c54e9e40ffe8157526ba6c0a6f2b98a54e906d0d32 currenttime:2018-10-27 22:55:07.209579837 +0800 CST m=+81.493323247
Processed block: 62d8caed59bded1648d680006dba481ac01e204d0bb6eadac925f69f3e0a6257, Chain height: 130, tipHash: 62d8caed59bded1648d680006dba481ac01e204d0bb6eadac925f69f3e0a6257, coinsTip hash: 62d8caed59bded1648d680006dba481ac01e204d0bb6eadac925f69f3e0a6257 currenttime:2018-10-27 22:55:07.22896996 +0800 CST m=+81.512713377
Processed block: 0b6943fef4fbfdaf15e6886eccaf9de7e856c72f0841d60191ca5825a8d67ecc, Chain height: 131, tipHash: 0b6943fef4fbfdaf15e6886eccaf9de7e856c72f0841d60191ca5825a8d67ecc, coinsTip has


-----------------------------------------

Processed block: 53fe66fad35bb02beedb855f3382811588e4465057d63a49a8b895b237a64996, Chain height: 100, tipHash: 53fe66fad35bb02beedb855f3382811588e4465057d63a49a8b895b237a64996, coinsTip hash: 53fe66fad35bb02beedb855f3382811588e4465057d63a49a8b895b237a64996 currenttime:2018-10-27 22:54:39.380722574 +0800 CST m=+53.664466003
> github.com/copernet/copernicus/logic/lblock.AcceptBlock() /home/j/.go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:265 (hits goroutine(790):1 total:1) (PC: 0xafb406)
Current event: 88589
   260:
   261:		// inDbp is nil indicate that this block haven't been write to disk
   262:		// when reindex, inDbp is not nil, and outDbp will be same as inDbp, and block will not be write to disk
   263:		outDbp, err = WriteBlockToDisk(bIndex, pblock, inDbp)
   264:		if err != nil {
=> 265:			panic("AcceptBlockHeader WriteBlockTo Disk err" + err.Error())
   266:		}
   267:
   268:		ReceivedBlockTransactions(pblock, bIndex, outDbp)
   269:		return
   270:	}
(dlv) locals
gChain = (*github.com/copernet/copernicus/model/chain.Chain)(0xc420149b80)
gPersist = (*github.com/copernet/copernicus/persist.PersistGlobal)(0xc4200ac000)
(dlv) args
bIndex = (*github.com/copernet/copernicus/model/blockindex.BlockIndex)(0xc420dae600)
err = error(github.com/copernet/copernicus/errcode.ProjectError) {Module: "", Code: 2001, Desc: "",...+1 more}
fNewBlock = (*bool)(0xc4200b71f6)
fRequested = true
inDbp = *github.com/copernet/copernicus/model/block.DiskBlockPos nil
outDbp = *github.com/copernet/copernicus/model/block.DiskBlockPos nil
pblock = (*github.com/copernet/copernicus/model/block.Block)(0xc420138fd0)
(dlv) bt
 0  0x0000000000afb406 in github.com/copernet/copernicus/logic/lblock.AcceptBlock
    at /home/j/.go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:265
 1  0x0000000000b4f9c9 in github.com/copernet/copernicus/service.ProcessNewBlock
    at /home/j/.go/src/github.com/copernet/copernicus/service/blockservice.go:75
 2  0x0000000000bce658 in github.com/copernet/copernicus/rpc.generateBlocks
    at /home/j/.go/src/github.com/copernet/copernicus/rpc/mining.go:593
 3  0x0000000000bce07c in github.com/copernet/copernicus/rpc.handleGenerate
    at /home/j/.go/src/github.com/copernet/copernicus/rpc/mining.go:543
 4  0x0000000000be4e34 in github.com/copernet/copernicus/rpc.(*Server).standardCmdResult
    at /home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:215
 5  0x0000000000be6995 in github.com/copernet/copernicus/rpc.(*Server).jsonRPCRead
    at /home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:375
 6  0x0000000000bea349 in github.com/copernet/copernicus/rpc.(*Server).Start.func1
    at /home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:444
 7  0x0000000000831734 in net/http.HandlerFunc.ServeHTTP
    at /usr/lib/go-1.10/src/net/http/server.go:1947
 8  0x0000000000834433 in net/http.(*ServeMux).ServeHTTP
    at /usr/lib/go-1.10/src/net/http/server.go:2340
 9  0x0000000000835814 in net/http.serverHandler.ServeHTTP
    at /usr/lib/go-1.10/src/net/http/server.go:2697
10  0x0000000000830722 in net/http.(*conn).serve
    at /usr/lib/go-1.10/src/net/http/server.go:1830
11  0x0000000000463ce1 in runtime.goexit
    at /usr/lib/go-1.10/src/runtime/asm_amd64.s:2361
(dlv) err


3.584555120
Processed block: 56c2d52b8c2f8e56c4d61fda1de8632104d4c10f48b589f3a603f2b2e0c5c15d, Chain height: 98, tipHash: 56c2d52b8c2f8e56c4d61fda1de8632104d4c10f48b589f3a603f2b2e0c5c15d, coinsTip hash: 56c2d52b8c2f8e56c4d61fda1de8632104d4c10f48b589f3a603f2b2e0c5c15d currenttime:2018-10-27 22:54:39.327007149 +0800 CST m=+53.610750524
Processed block: 6bf42ca46fbca3f6bc30073340cd706bd223f2151c578efe7eb6d49824e1ea7c, Chain height: 99, tipHash: 6bf42ca46fbca3f6bc30073340cd706bd223f2151c578efe7eb6d49824e1ea7c, coinsTip hash: 6bf42ca46fbca3f6bc30073340cd706bd223f2151c578efe7eb6d49824e1ea7c currenttime:2018-10-27 22:54:39.354403075 +0800 CST m=+53.638146471
Processed block: 53fe66fad35bb02beedb855f3382811588e4465057d63a49a8b895b237a64996, Chain height: 100, tipHash: 53fe66fad35bb02beedb855f3382811588e4465057d63a49a8b895b237a64996, coinsTip hash: 53fe66fad35bb02beedb855f3382811588e4465057d63a49a8b895b237a64996 currenttime:2018-10-27 22:54:39.380722574 +0800 CST m=+53.664466003
> github.com/copernet/copernicus/logic/lblock.AcceptBlock() /home/j/.go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:265 (hits goroutine(790):1 total:1) (PC: 0xafb406)
Current event: 88589
   260:
   261:		// inDbp is nil indicate that this block haven't been write to disk
   262:		// when reindex, inDbp is not nil, and outDbp will be same as inDbp, and block will not be write to disk
   263:		outDbp, err = WriteBlockToDisk(bIndex, pblock, inDbp)
   264:		if err != nil {
=> 265:			panic("AcceptBlockHeader WriteBlockTo Disk err" + err.Error())
   266:		}
   267:
   268:		ReceivedBlockTransactions(pblock, bIndex, outDbp)
   269:		return
   270:	}
(dlv) locals
gChain = (*github.com/copernet/copernicus/model/chain.Chain)(0xc420149b80)
gPersist = (*github.com/copernet/copernicus/persist.PersistGlobal)(0xc4200ac000)
(dlv) args
bIndex = (*github.com/copernet/copernicus/model/blockindex.BlockIndex)(0xc420dae600)
err = error(github.com/copernet/copernicus/errcode.ProjectError) {Module: "", Code: 2001, Desc: "",...+1 more}
fNewBlock = (*bool)(0xc4200b71f6)
fRequested = true
inDbp = *github.com/copernet/copernicus/model/block.DiskBlockPos nil
outDbp = *github.com/copernet/copernicus/model/block.DiskBlockPos nil
pblock = (*github.com/copernet/copernicus/model/block.Block)(0xc420138fd0)
(dlv) bt
 0  0x0000000000afb406 in github.com/copernet/copernicus/logic/lblock.AcceptBlock
    at /home/j/.go/src/github.com/copernet/copernicus/logic/lblock/lblock.go:265
 1  0x0000000000b4f9c9 in github.com/copernet/copernicus/service.ProcessNewBlock
    at /home/j/.go/src/github.com/copernet/copernicus/service/blockservice.go:75
 2  0x0000000000bce658 in github.com/copernet/copernicus/rpc.generateBlocks
    at /home/j/.go/src/github.com/copernet/copernicus/rpc/mining.go:593
 3  0x0000000000bce07c in github.com/copernet/copernicus/rpc.handleGenerate
    at /home/j/.go/src/github.com/copernet/copernicus/rpc/mining.go:543
 4  0x0000000000be4e34 in github.com/copernet/copernicus/rpc.(*Server).standardCmdResult
    at /home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:215
 5  0x0000000000be6995 in github.com/copernet/copernicus/rpc.(*Server).jsonRPCRead
    at /home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:375
 6  0x0000000000bea349 in github.com/copernet/copernicus/rpc.(*Server).Start.func1
    at /home/j/.go/src/github.com/copernet/copernicus/rpc/rpcserver.go:444
 7  0x0000000000831734 in net/http.HandlerFunc.ServeHTTP
    at /usr/lib/go-1.10/src/net/http/server.go:1947
 8  0x0000000000834433 in net/http.(*ServeMux).ServeHTTP
    at /usr/lib/go-1.10/src/net/http/server.go:2340
 9  0x0000000000835814 in net/http.serverHandler.ServeHTTP
    at /usr/lib/go-1.10/src/net/http/server.go:2697
10  0x0000000000830722 in net/http.(*conn).serve
    at /usr/lib/go-1.10/src/net/http/server.go:1830
11  0x0000000000463ce1 in runtime.goexit
    at /usr/lib/go-1.10/src/runtime/asm_amd64.s:2361
(dlv) err
Command failed: command not available
(dlv) p err
error(github.com/copernet/copernicus/errcode.ProjectError) {
	Module: "",
	Code: 2001,
	Desc: "",
	ErrorCode: fmt.Stringer nil,}
(dlv) b disk.go:304
Breakpoint 2 set at 0xad5daa for github.com/copernet/copernicus/persist/disk.WriteBlockToDisk() /home/j/.go/src/github.com/copernet/copernicus/persist/disk/disk.go:304
(dlv) rw
> github.com/copernet/copernicus/persist/disk.WriteBlockToDisk() /home/j/.go/src/github.com/copernet/copernicus/persist/disk/disk.go:304 (hits goroutine(790):1 total:1) (PC: 0xad5daa)
Current event: 88589
   299:			return false
   300:		}
   301:		lenData := lenBuf.Bytes()
   302:		_, err = file.Write(lenData)
   303:		if err != nil {
=> 304:			log.Error("WriteBlockToDisk: write lenData error: %v", err)
   305:			return false
   306:		}
   307:		_, err = file.Write(buf.Bytes())
   308:		if err != nil {
   309:			log.Error("WriteBlockToDisk: write buf.Bytes() error: %v", err)
(dlv) b disk.go:402
Command failed: could not find /home/j/.go/src/github.com/copernet/copernicus/persist/disk/disk.go:402
(dlv) b disk.go:302
Breakpoint 3 set at 0xad5d2e for github.com/copernet/copernicus/persist/disk.WriteBlockToDisk() /home/j/.go/src/github.com/copernet/copernicus/persist/disk/disk.go:302
(dlv) rw
> github.com/copernet/copernicus/persist/disk.WriteBlockToDisk() /home/j/.go/src/github.com/copernet/copernicus/persist/disk/disk.go:302 (hits goroutine(790):1 total:1) (PC: 0xad5d2e)
Current event: 88589
   297:		if err != nil {
   298:			log.Error("WriteBlockToDisk: put size error: %v", err)
   299:			return false
   300:		}
   301:		lenData := lenBuf.Bytes()
=> 302:		_, err = file.Write(lenData)
   303:		if err != nil {
   304:			log.Error("WriteBlockToDisk: write lenData error: %v", err)
   305:			return false
   306:		}
   307:		_, err = file.Write(buf.Bytes())
(dlv) p lenData
[]uint8 len: 4, cap: 64, [152,0,0,0]
(dlv) p size
152
(dlv) step
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:140 (PC: 0x5075d3)
Current event: 88589
   135:	}
   136:
   137:	// Write writes len(b) bytes to the File.
   138:	// It returns the number of bytes written and an error, if any.
   139:	// Write returns a non-nil error when n != len(b).
=> 140:	func (f *File) Write(b []byte) (n int, err error) {
   141:		if err := f.checkValid("write"); err != nil {
   142:			return 0, err
   143:		}
   144:		n, e := f.write(b)
   145:		if n < 0 {
(dlv) n
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:141 (PC: 0x5075fb)
Current event: 88589
   136:
   137:	// Write writes len(b) bytes to the File.
   138:	// It returns the number of bytes written and an error, if any.
   139:	// Write returns a non-nil error when n != len(b).
   140:	func (f *File) Write(b []byte) (n int, err error) {
=> 141:		if err := f.checkValid("write"); err != nil {
   142:			return 0, err
   143:		}
   144:		n, e := f.write(b)
   145:		if n < 0 {
   146:			n = 0
(dlv)
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:144 (PC: 0x50766f)
Current event: 88589
   139:	// Write returns a non-nil error when n != len(b).
   140:	func (f *File) Write(b []byte) (n int, err error) {
   141:		if err := f.checkValid("write"); err != nil {
   142:			return 0, err
   143:		}
=> 144:		n, e := f.write(b)
   145:		if n < 0 {
   146:			n = 0
   147:		}
   148:		if n != len(b) {
   149:			err = io.ErrShortWrite
(dlv)
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:145 (PC: 0x5076ee)
Current event: 88589
   140:	func (f *File) Write(b []byte) (n int, err error) {
   141:		if err := f.checkValid("write"); err != nil {
   142:			return 0, err
   143:		}
   144:		n, e := f.write(b)
=> 145:		if n < 0 {
   146:			n = 0
   147:		}
   148:		if n != len(b) {
   149:			err = io.ErrShortWrite
   150:		}
(dlv) p e
error(syscall.Errno) github.com/copernet/copernicus/vendor/golang.org/x/sys/unix.ENOMEM
(dlv) args
b = []uint8 len: 4, cap: 64, [...]
err = error nil
f = (*os.File)(0xc420e40528)
n = 0
(dlv) p b
[]uint8 len: 4, cap: 64, [152,0,0,0]
(dlv) p f
*os.File {
	file: *os.file {
		pfd: (*internal/poll.FD)(0xc420d62640),
		name: "/media/psf/bj1809120051/lab/walleubuntu/test/cache/node0/regtest...+20 more",
		dirinfo: *os.dirInfo nil,
		nonblock: false,
		stdoutOrErr: false,},}
(dlv) list
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:145 (PC: 0x5076ee)
Current event: 88589
   140:	func (f *File) Write(b []byte) (n int, err error) {
   141:		if err := f.checkValid("write"); err != nil {
   142:			return 0, err
   143:		}
   144:		n, e := f.write(b)
=> 145:		if n < 0 {
   146:			n = 0
   147:		}
   148:		if n != len(b) {
   149:			err = io.ErrShortWrite
   150:		}
(dlv) b file.go:144
Command failed: Location "file.go:144" ambiguous: /home/j/.go/src/github.com/copernet/copernicus/vendor/github.com/astaxie/beego/logs/file.go, /home/j/.go/src/github.com/copernet/copernicus/vendor/github.com/spf13/afero/mem/file.go, /usr/lib/go-1.10/src/os/file.go…
(dlv) b /usr/lib/go-1.10/src/os/file.go:145
Breakpoint 4 set at 0x5076ee for os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:145
(dlv) list
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:145 (PC: 0x5076ee)
Current event: 88589
   140:	func (f *File) Write(b []byte) (n int, err error) {
   141:		if err := f.checkValid("write"); err != nil {
   142:			return 0, err
   143:		}
   144:		n, e := f.write(b)
=> 145:		if n < 0 {
   146:			n = 0
   147:		}
   148:		if n != len(b) {
   149:			err = io.ErrShortWrite
   150:		}
(dlv) n
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:148 (PC: 0x50770e)
Current event: 88589
   143:		}
   144:		n, e := f.write(b)
   145:		if n < 0 {
   146:			n = 0
   147:		}
=> 148:		if n != len(b) {
   149:			err = io.ErrShortWrite
   150:		}
   151:
   152:		epipecheck(f, e)
   153:
(dlv) p b
[]uint8 len: 4, cap: 64, [152,0,0,0]
(dlv) n
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:149 (PC: 0x50772d)
Current event: 88589
   144:		n, e := f.write(b)
   145:		if n < 0 {
   146:			n = 0
   147:		}
   148:		if n != len(b) {
=> 149:			err = io.ErrShortWrite
   150:		}
   151:
   152:		epipecheck(f, e)
   153:
   154:		if e != nil {
(dlv)
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:152 (PC: 0x50774d)
Current event: 88589
   147:		}
   148:		if n != len(b) {
   149:			err = io.ErrShortWrite
   150:		}
   151:
=> 152:		epipecheck(f, e)
   153:
   154:		if e != nil {
   155:			err = f.wrapErr("write", e)
   156:		}
   157:
(dlv) step
> os.epipecheck() /usr/lib/go-1.10/src/os/file_unix.go:146 (PC: 0x50a7d3)
Current event: 88589
   141:	}
   142:
   143:	// epipecheck raises SIGPIPE if we get an EPIPE error on standard
   144:	// output or standard error. See the SIGPIPE docs in os/signal, and
   145:	// issue 11845.
=> 146:	func epipecheck(file *File, e error) {
   147:		if e == syscall.EPIPE && file.stdoutOrErr {
   148:			sigpipe()
   149:		}
   150:	}
   151:
(dlv) n
> os.epipecheck() /usr/lib/go-1.10/src/os/file_unix.go:147 (PC: 0x50a7df)
Current event: 88589
   142:
   143:	// epipecheck raises SIGPIPE if we get an EPIPE error on standard
   144:	// output or standard error. See the SIGPIPE docs in os/signal, and
   145:	// issue 11845.
   146:	func epipecheck(file *File, e error) {
=> 147:		if e == syscall.EPIPE && file.stdoutOrErr {
   148:			sigpipe()
   149:		}
   150:	}
   151:
   152:	// DevNull is the name of the operating system's ``null device.''
(dlv)
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:154 (PC: 0x507772)
Values returned:

Current event: 88589
   149:			err = io.ErrShortWrite
   150:		}
   151:
   152:		epipecheck(f, e)
   153:
=> 154:		if e != nil {
   155:			err = f.wrapErr("write", e)
   156:		}
   157:
   158:		return n, err
   159:	}
(dlv)
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:155 (PC: 0x50777e)
Current event: 88589
   150:		}
   151:
   152:		epipecheck(f, e)
   153:
   154:		if e != nil {
=> 155:			err = f.wrapErr("write", e)
   156:		}
   157:
   158:		return n, err
   159:	}
   160:
(dlv) p e
error(syscall.Errno) github.com/copernet/copernicus/vendor/golang.org/x/sys/unix.ENOMEM
(dlv)
Command failed: not enough arguments
(dlv)
Command failed: not enough arguments
(dlv) list
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:155 (PC: 0x50777e)
Current event: 88589
   150:		}
   151:
   152:		epipecheck(f, e)
   153:
   154:		if e != nil {
=> 155:			err = f.wrapErr("write", e)
   156:		}
   157:
   158:		return n, err
   159:	}
   160:
(dlv) n
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:158 (PC: 0x5077d4)
Current event: 88589
   153:
   154:		if e != nil {
   155:			err = f.wrapErr("write", e)
   156:		}
   157:
=> 158:		return n, err
   159:	}
   160:
   161:	// WriteAt writes len(b) bytes to the File starting at byte offset off.
   162:	// It returns the number of bytes written and an error, if any.
   163:	// WriteAt returns a non-nil error when n != len(b).
(dlv) p err
error(*os.PathError) *{
	Op: "write",
	Path: "/media/psf/bj1809120051/lab/walleubuntu/test/cache/node0/regtest...+20 more",
	Err: error(syscall.Errno) github.com/copernet/copernicus/vendor/golang.org/x/sys/unix.ENOMEM,}
(dlv)
Command failed: not enough arguments
(dlv)
Command failed: not enough arguments
(dlv) lis
Command failed: command not available
(dlv) list
> os.(*File).Write() /usr/lib/go-1.10/src/os/file.go:158 (PC: 0x5077d4)
Current event: 88589
   153:
   154:		if e != nil {
   155:			err = f.wrapErr("write", e)
   156:		}
   157:
=> 158:		return n, err
   159:	}
   160:
   161:	// WriteAt writes len(b) bytes to the File starting at byte offset off.
   162:	// It returns the number of bytes written and an error, if any.
   163:	// WriteAt returns a non-nil error when n != len(b).
(dlv) c
> github.com/copernet/copernicus/persist/disk.WriteBlockToDisk() /home/j/.go/src/github.com/copernet/copernicus/persist/disk/disk.go:304 (hits goroutine(790):2 total:2) (PC: 0xad5daa)
Current event: 88589
Warning: listing may not match stale executable
   299:			log.Error("WriteBlockToDisk: put size error: %v", err)
   300:			return false
   301:		}
   302:
   303:		lenData := lenBuf.Bytes()
=> 304:		_, err = file.Write(lenData)
   305:		if err != nil {
   306:			log.Error("WriteBlockToDisk: write lenData error: %v", err)
   307:			return false
   308:		}
   309:
(dlv) p pos
*github.com/copernet/copernicus/model/block.DiskBlockPos {File: 0, Pos: 18385}
(dlv) args
block = (*github.com/copernet/copernicus/model/block.Block)(0xc420138fd0)
pos = (*github.com/copernet/copernicus/model/block.DiskBlockPos)(0xc4201f0468)
~r2 = false
(dlv) p file
*os.File {
	file: *os.file {
		pfd: (*internal/poll.FD)(0xc420d62640),
		name: "/media/psf/bj1809120051/lab/walleubuntu/test/cache/node0/regtest...+20 more",
		dirinfo: *os.dirInfo nil,
		nonblock: false,
		stdoutOrErr: false,},}
(dlv) list
> github.com/copernet/copernicus/persist/disk.WriteBlockToDisk() /home/j/.go/src/github.com/copernet/copernicus/persist/disk/disk.go:304 (hits goroutine(790):2 total:2) (PC: 0xad5daa)
Current event: 88589
Warning: listing may not match stale executable
   299:			log.Error("WriteBlockToDisk: put size error: %v", err)
   300:			return false
   301:		}
   302:
   303:		lenData := lenBuf.Bytes()
=> 304:		_, err = file.Write(lenData)
   305:		if err != nil {
   306:			log.Error("WriteBlockToDisk: write lenData error: %v", err)
   307:			return false
   308:		}
   309:
(dlv) buf
Command failed: command not available
(dlv) p buf
*bytes.Buffer {
	buf: []uint8 len: 152, cap: 160, [1,0,0,0,103,130,106,116,181,241,80,186,167,254,71,21,46,37,35,78,220,132,129,208,241,70,223,12,30,217,178,170,49,206,147,15,10,126,54,171,128,157,102,44,141,211,182,198,80,151,206,22,29,9,17,56,160,169,17,125,115,230,138,222,...+88 more],
	off: 0,
	bootstrap: [64]uint8 [1,0,0,0,103,130,106,116,181,241,80,186,167,254,71,21,46,37,35,78,220,132,129,208,241,70,223,12,30,217,178,170,49,206,147,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
	lastRead: opInvalid,}
(dlv)
Command failed: not enough arguments
(dlv)
Command failed: not enough arguments
(dlv)


node will panic when running simple undo testcase

Describe the bug
when running testcase for undo, the node will panic.

To Reproduce
Steps to reproduce the behavior:

  1. running testcase single_node_undo.py
pipenv run python single_node_undo.py --tracerpc -l DEBUG --nocleanup

Expected behavior
undo successfully and no panic

Additional context
panic logs:

2018-10-18 12:50:29.269000 TestFramework (INFO): height 101, hash: 12158721378675693663521002650064272439080797260410274965317733538595202393199
2018-10-18 12:50:29.269000 TestFramework (INFO): #5. expect node0 switch to new chain
panic: newcoin fresh and oldcoin has spent

goroutine 148 [running]:
github.com/copernet/copernicus/model/utxo.(*CoinsLruCache).UpdateCoins(0xc000196080, 0xc0001d2520, 0xc000744800, 0x0, 0x0)
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/model/utxo/lrucache.go:125 +0x863
github.com/copernet/copernicus/logic/lchain.ConnectTip(0xc000f0f600, 0xc0001fa000, 0xc000ecb1a0, 0x0, 0x0)
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/logic/lchain/lchain.go:256 +0x7f6
github.com/copernet/copernicus/logic/lchain.ActivateBestChainStep(0xc000f0f600, 0xc0001fa000, 0xc000ecb0e6, 0xc000ecb1a0, 0x0, 0x0)
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/logic/lchain/lactivechain.go:155 +0x404
github.com/copernet/copernicus/logic/lchain.ActivateBestChain(0xc0001fa000, 0x0, 0x0)
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/logic/lchain/lactivechain.go:71 +0x2d5
github.com/copernet/copernicus/service.ProcessNewBlock(0xc0001fa000, 0x4a88301, 0xc000ecb4d3, 0x0, 0x0)
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/service/blockservice.go:89 +0x5ef
github.com/copernet/copernicus/service.ProcessBlock(0xc0001fa000, 0xc0002e3e00, 0x0, 0x0)
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/service/blockservice.go:43 +0x36e
github.com/copernet/copernicus/net/syncmanager.(*SyncManager).handleBlockMsg(0xc00019a0c0, 0xc000e321e0)
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:607 +0x333
github.com/copernet/copernicus/net/syncmanager.(*SyncManager).messagesHandler(0xc00019a0c0)
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:1179 +0x67f
created by github.com/copernet/copernicus/net/syncmanager.(*SyncManager).Start
	/Users/bj1809120051/.go/src/github.com/copernet/copernicus/net/syncmanager/syncmanager.go:1426 +0xb8
2018-10-18 12:50:29.366000 TestFramework.mininode (DEBUG): Closing connection to: 127.0.0.1:12864
2018-10-18 12:50:29.366000 TestFramework.mininode (DEBUG): Closing connection to: 127.0.0.1:12864
2018-10-18 12:50:29.366000 TestFramework.mininode (DEBUG): Network thread closing

p2p-fullblocktest.py report a crash

Describe the bug
p2p-fullblocktest.py has bug tx panic: the transaction children set is different .. lmempool.CheckMempool.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

services flag is not displayed correctly

Describe the bug
services flag is not displayed correctly in rpc getpeerinfo and getnetworkinfo

To Reproduce
Steps to reproduce the behavior:

  1. coperctl getnetworkinfo
  2. coperctl getpeerinfo

Expected behavior
{
"version": 6,
"subversion": "/Copernicus:0.0.6(EB32.0)/",
"protocolversion": 70013,
"localservices": "0000000000000011",
...
}

Screenshots
If applicable, add screenshots to help explain your problem.
image

Desktop (please complete the following information):

  • OS: linux

panic: open /root/.bitcoincash/bitcoincash.yml, get GOPATH failed

Describe the bug

root@iZm5eakzlq9sm4ycm9lw0wZ:/home/whunmr/copernicus# cat nohup.out
get GOPATH failed, please check if gopath is configured.
panic: open /root/.bitcoincash/bitcoincash.yml: no such file or directory

goroutine 1 [running]:
github.com/copernet/copernicus/conf.must(0xd71900, 0x0, 0xe1efe0, 0xc4201ff4a0, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/conf/conf.go:344 +0x9a
github.com/copernet/copernicus/conf.InitConfig(0xc42001e1e0, 0x1, 0x1, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/conf/conf.go:254 +0xdc2
main.appInitMain(0xc42001e1e0, 0x1, 0x1)
	/home/j/.go/src/github.com/copernet/copernicus/initmain.go:27 +0x5d
main.bchMain(0xe23d00, 0xc4200200b8, 0xc42001e1e0, 0x1, 0x1, 0x0, 0x0)
	/home/j/.go/src/github.com/copernet/copernicus/main.go:38 +0x68
main.main()
	/home/j/.go/src/github.com/copernet/copernicus/main.go:107 +0x1ef

To Reproduce
run copernicus on server without gopath configured.

Expected behavior
run normally, without depends on gopath

panic: init CoinsDB failed...resource temporarily unavailable

An error occurred while running the client :

panic: init CoinsDB failed...resource temporarily unavailable

goroutine 1 [running]:
github.com/copernet/copernicus/model/utxo.newCoinsDB(0xc420280340, 0xc420204340)
       copernet/copernicus/model/utxo/coindb.go:132 +0x128
github.com/copernet/copernicus/model/utxo.InitUtxoLruTip(0xc420155e70)
       copernet/copernicus/model/utxo/lrucache.go:30 +0x2e
main.appInitMain(0xc42001e1b0, 0x1, 0x1)
        copernet/copernicus/initmain.go:28 +0x1a1
main.bchMain(0x4785700, 0xc420026140, 0xc42001e1b0, 0x1, 0x1, 0x0, 0x0)
        copernet/copernicus/main.go:37 +0x4b
main.main()
       copernet/copernicus/main.go:101 +0x103

The reason is that the client did not close properly , The client needs to be monitored and run in the background.

QuickStart issue

OS

mac

occurred problem

when I execute ./autoinstall.sh, the following error occurred:

./autoinstall.sh: line 3: autoreconf: command not found

autoinstall.sh file:

  1 #!/bin/sh
  2 set -e
  3 autoreconf -if --warnings=all
  4 ./configure --enable-module-ecdh --enable-module-recovery --enable-experimental
  5 make -j8
  6 make install

identify the problem

we don't have the dependencies we need to install

solution

brew install automake berkeley-db libtool boost --c++11 miniupnpc openssl pkg-config protobuf --c++11 libevent

[cmpctblock]In order to achieve quicker block relay, we need to send newValidPoWBlock through cmpctblock msg after block accepted

describe
currently block relay is throught inv/headers, getdata cycle

Describe the solution you'd like
send newValidPoWBlock through cmpctblock message

Additional context

refer to logic in ABC's code:

void PeerLogicValidation::NewPoWValidBlock(
    const CBlockIndex *pindex, const std::shared_ptr<const CBlock> &pblock) {

    connman->ForEachNode([this, &pcmpctblock, pindex, &msgMaker,
                          &hashBlock](CNode *pnode) {

        // If the peer has, or we announced to them the previous block already,
        // but we don't think they have this one, go ahead and announce it.
        if (state.fPreferHeaderAndIDs && !PeerHasHeader(&state, pindex) &&
            PeerHasHeader(&state, pindex->pprev)) {

            LogPrint(BCLog::NET, "%s sending header-and-ids %s to peer=%d\n",
                     "PeerLogicValidation::NewPoWValidBlock",
                     hashBlock.ToString(), pnode->GetId());
            connman->PushMessage(
                pnode, msgMaker.Make(NetMsgType::CMPCTBLOCK, *pcmpctblock));
            state.pindexBestHeaderSent = pindex;
        }
    });
}

    else if (strCommand == NetMsgType::GETHEADERS) {
        ...
        // pindex can be nullptr either if we sent chainActive.Tip() OR
        // if our peer has chainActive.Tip() (and thus we are sending an empty
        // headers message). In both cases it's safe to update
        // pindexBestHeaderSent to be our tip.
        //
        // It is important that we simply reset the BestHeaderSent value here,
        // and not max(BestHeaderSent, newHeaderSent). We might have announced
        // the currently-being-connected tip using a compact block, which
        // resulted in the peer sending a headers request, which we respond to
        // without the new block. By resetting the BestHeaderSent, we ensure we
        // will re-announce the new block via headers (or compact blocks again)
        // in the SendMessages logic.
        nodestate->pindexBestHeaderSent = pindex ? pindex : chainActive.Tip();
        connman->PushMessage(pfrom,
                             msgMaker.Make(NetMsgType::HEADERS, vHeaders));
    }

hang on download headers during IBD

Describe the bug
During IBD, haven't start download block in nearly one hour, seems still syncing headers.

root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 13:29:09 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 13:35:04 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1245077,
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 14:18:12 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo
{
  "chain": "test",
  "blocks": 0,
  "headers": 1245077,
  "bestblockhash": "000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943",

root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 14:18:56 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus#

To Reproduce
running instance from master code on testnet: 3a07937

Expected behavior
quickly download the headers of all block.
when peer hang we should switch to other peers to download the headers.

Additional context

root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 13:29:09 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 13:35:04 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1235077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1241077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1241077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1241077,
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 13:59:41 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1241077,
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1241077,
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 14:05:42 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 14:09:36 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1241077,
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 14:09:41 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo
{
  "chain": "test",
  "blocks": 0,
  "headers": 1241077,
  "bestblockhash": "000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943",
  "difficulty": 1,
  "mediantime": 1296688602,
  "verificationprogress": 4.552473572222998e-8,
  "pruned": false,
  "chainwork": "0000000000000000000000000000000000000000000000000000000100010001",
  "softforks": [
    {
      "id": "bip34",
      "version": 2,
      "reject": {
        "status": false
      }
    },
    {
      "id": "bip66",
      "version": 3,
      "reject": {
        "status": false
      }
    },
    {
      "id": "bip65",
      "version": 4,
      "reject": {
        "status": false
      }
    },
    {
      "id": "csv",
      "version": 5,
      "reject": {
        "status": false
      }
    }
  ],
  "bip9_softforks": {
    "csv": {
      "status": "defined",
      "bit": 0,
      "startTime": 1456790400,
      "timeout": 1493596800,
      "since": 0
    },
    "testdummy": {
      "status": "defined",
      "bit": 28,
      "startTime": 1199145601,
      "timeout": 1230767999,
      "since": 0
    }
  }
}
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getpeerinfo
[
  {
    "id": 40,
    "addr": "18.219.186.84:18333",
    "addrlocal": "114.215.41.16:55222",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 5274,
    "bytesrecv": 491801,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 272255,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.2(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 43,
    "addr": "207.154.196.149:18333",
    "addrlocal": "114.215.41.16:44628",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 2133,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 336389,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.2(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 4,
    "addr": "118.26.129.214:18333",
    "addrlocal": "114.215.41.16:54138",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 2078,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 23060,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.2(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 33,
    "addr": "144.217.73.86:18333",
    "addrlocal": "114.215.41.16:48664",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5806,
    "bytesrecv": 493971,
    "conntime": 1544419163,
    "timeoffset": 6,
    "pingtime": 286042,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.3(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 47,
    "addr": "35.228.248.211:18333",
    "addrlocal": "114.215.41.16:51166",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 3564,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 332963,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 58,
    "addr": "35.234.158.44:18333",
    "addrlocal": "114.215.41.16:44736",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422286,
    "lastrecv": 1544422286,
    "bytessent": 1880,
    "bytesrecv": 3564,
    "conntime": 1544419164,
    "timeoffset": 0,
    "pingtime": 304040,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 31,
    "addr": "18.202.113.209:18333",
    "addrlocal": "114.215.41.16:38730",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5196,
    "bytesrecv": 491511,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 285493,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.1(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 32,
    "addr": "18.202.113.209:18333",
    "addrlocal": "114.215.41.16:38770",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5135,
    "bytesrecv": 491341,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 290618,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.1(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 38,
    "addr": "52.214.195.71:18333",
    "addrlocal": "114.215.41.16:45834",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 2046,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 298651,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.0(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 39,
    "addr": "34.253.51.235:18333",
    "addrlocal": "114.215.41.16:54250",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5257,
    "bytesrecv": 491747,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 306759,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.4(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 3,
    "addr": "47.99.176.58:18333",
    "addrlocal": "114.215.41.16:40482",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5135,
    "bytesrecv": 491256,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 24233,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.2(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 16,
    "addr": "159.89.149.75:18333",
    "addrlocal": "114.215.41.16:38432",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 1969,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 186852,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.17.1(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1268000,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 26,
    "addr": "35.237.217.145:18333",
    "addrlocal": "114.215.41.16:41872",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 3619,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 226133,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 30,
    "addr": "70.36.125.75:18333",
    "addrlocal": "114.215.41.16:35344",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422286,
    "lastrecv": 1544422286,
    "bytessent": 5135,
    "bytesrecv": 491281,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 174826,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.4(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 23,
    "addr": "67.205.179.161:18333",
    "addrlocal": "114.215.41.16:33258",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422283,
    "lastrecv": 1544422283,
    "bytessent": 1880,
    "bytesrecv": 2078,
    "conntime": 1544419163,
    "timeoffset": 1,
    "pingtime": 230441,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.0(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 27,
    "addr": "67.205.179.161:18333",
    "addrlocal": "114.215.41.16:33324",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 1913,
    "conntime": 1544419163,
    "timeoffset": 1,
    "pingtime": 227244,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.0(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 55,
    "addr": "18.231.178.141:18333",
    "addrlocal": "114.215.41.16:46610",
    "services": "0000000000000037",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 1895,
    "conntime": 1544419163,
    "timeoffset": -1,
    "pingtime": 755235,
    "version": 70015,
    "subver": "/Bitcoin XT:0.11.0K(Linux; x86_64; EB32)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1266533,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 48,
    "addr": "52.214.195.71:18333",
    "addrlocal": "114.215.41.16:45852",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 1936,
    "conntime": 1544419163,
    "timeoffset": -1,
    "pingtime": 288564,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.0(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 72,
    "addr": "52.25.222.97:18333",
    "addrlocal": "114.215.41.16:45312",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422211,
    "lastrecv": 1544422211,
    "bytessent": 3858,
    "bytesrecv": 328746,
    "conntime": 1544419449,
    "timeoffset": 0,
    "pingtime": 205674,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.4(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273285,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 8,
    "addr": "39.104.60.222:18333",
    "addrlocal": "114.215.41.16:60748",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5274,
    "bytesrecv": 491541,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 58539,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.5(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 71,
    "addr": "199.192.20.83:18333",
    "addrlocal": "114.215.41.16:43226",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422293,
    "lastrecv": 1544422293,
    "bytessent": 1816,
    "bytesrecv": 1969,
    "conntime": 1544419291,
    "timeoffset": -1,
    "pingtime": 201750,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.17.1(EB32.0; bitcore)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1268000,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 14,
    "addr": "3.0.44.122:18333",
    "addrlocal": "114.215.41.16:59858",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422287,
    "lastrecv": 1544422287,
    "bytessent": 5135,
    "bytesrecv": 491181,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 136776,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.2(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 42,
    "addr": "188.40.93.205:18333",
    "addrlocal": "114.215.41.16:57170",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 1969,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 315644,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.17.1(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1268000,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 45,
    "addr": "35.228.248.211:18333",
    "addrlocal": "114.215.41.16:51068",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 3509,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 332813,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 52,
    "addr": "18.231.178.141:18333",
    "addrlocal": "114.215.41.16:46618",
    "services": "0000000000000037",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 1950,
    "conntime": 1544419163,
    "timeoffset": -1,
    "pingtime": 727530,
    "version": 70015,
    "subver": "/Bitcoin XT:0.11.0K(Linux; x86_64; EB32)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1266533,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 59,
    "addr": "203.162.80.101:18333",
    "addrlocal": "114.215.41.16:46430",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422286,
    "lastrecv": 1544422286,
    "bytessent": 5135,
    "bytesrecv": 491231,
    "conntime": 1544419164,
    "timeoffset": 0,
    "pingtime": 95462,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.0(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 65,
    "addr": "52.44.250.161:18333",
    "addrlocal": "114.215.41.16:48332",
    "services": "0000000000000021",
    "relaytxes": true,
    "lastsend": 1544422259,
    "lastrecv": 1544422259,
    "bytessent": 1816,
    "bytesrecv": 3044,
    "conntime": 1544419258,
    "timeoffset": 0,
    "pingtime": 345063,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.15.1(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1196428,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 10,
    "addr": "54.250.148.121:18333",
    "addrlocal": "114.215.41.16:39568",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5135,
    "bytesrecv": 491231,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 92778,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.2(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 19,
    "addr": "35.237.217.145:18333",
    "addrlocal": "114.215.41.16:41902",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 3564,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 223009,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 35,
    "addr": "138.68.231.122:18333",
    "addrlocal": "114.215.41.16:42554",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 2274,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 182756,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.5(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1268003,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 37,
    "addr": "144.217.73.86:18333",
    "addrlocal": "114.215.41.16:48630",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 6006,
    "bytesrecv": 493627,
    "conntime": 1544419163,
    "timeoffset": 6,
    "pingtime": 275821,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.3(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 70,
    "addr": "116.125.120.98:18333",
    "addrlocal": "114.215.41.16:53864",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422291,
    "lastrecv": 1544422291,
    "bytessent": 1816,
    "bytesrecv": 2990,
    "conntime": 1544419291,
    "timeoffset": 0,
    "pingtime": 44053,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.4(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1266534,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 7,
    "addr": "39.104.60.222:18333",
    "addrlocal": "114.215.41.16:60794",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5135,
    "bytesrecv": 491286,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 52380,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.5(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 17,
    "addr": "35.237.217.145:18333",
    "addrlocal": "114.215.41.16:41880",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 3509,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 221844,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 24,
    "addr": "67.205.179.161:18333",
    "addrlocal": "114.215.41.16:33250",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 1968,
    "conntime": 1544419163,
    "timeoffset": 1,
    "pingtime": 228926,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.0(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 51,
    "addr": "18.231.178.141:18333",
    "addrlocal": "114.215.41.16:46574",
    "services": "0000000000000037",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 1895,
    "conntime": 1544419163,
    "timeoffset": -1,
    "pingtime": 731421,
    "version": 70015,
    "subver": "/Bitcoin XT:0.11.0K(Linux; x86_64; EB32)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1266533,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 2,
    "addr": "118.26.129.214:18333",
    "addrlocal": "114.215.41.16:54140",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 1968,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 20294,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.2(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 53,
    "addr": "18.231.178.141:18333",
    "addrlocal": "114.215.41.16:46586",
    "services": "0000000000000037",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 1895,
    "conntime": 1544419163,
    "timeoffset": -1,
    "pingtime": 736215,
    "version": 70015,
    "subver": "/Bitcoin XT:0.11.0K(Linux; x86_64; EB32)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1266533,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 54,
    "addr": "18.231.178.141:18333",
    "addrlocal": "114.215.41.16:46598",
    "services": "0000000000000037",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 2005,
    "conntime": 1544419163,
    "timeoffset": -1,
    "pingtime": 746108,
    "version": 70015,
    "subver": "/Bitcoin XT:0.11.0K(Linux; x86_64; EB32)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1266533,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 62,
    "addr": "35.204.58.182:18333",
    "addrlocal": "114.215.41.16:56696",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422305,
    "lastrecv": 1544422305,
    "bytessent": 1880,
    "bytesrecv": 3509,
    "conntime": 1544419166,
    "timeoffset": -1,
    "pingtime": 304719,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 22,
    "addr": "35.237.217.145:18333",
    "addrlocal": "114.215.41.16:41956",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 3564,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 223333,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 28,
    "addr": "35.237.217.145:18333",
    "addrlocal": "114.215.41.16:41940",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 3509,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 232583,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 44,
    "addr": "207.154.196.149:18333",
    "addrlocal": "114.215.41.16:44700",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 1968,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 324319,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.2(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 61,
    "addr": "35.204.58.182:18333",
    "addrlocal": "114.215.41.16:56654",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422294,
    "lastrecv": 1544422294,
    "bytessent": 1880,
    "bytesrecv": 3564,
    "conntime": 1544419170,
    "timeoffset": 0,
    "pingtime": 302942,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 49,
    "addr": "52.214.195.71:18333",
    "addrlocal": "114.215.41.16:45854",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 1936,
    "conntime": 1544419163,
    "timeoffset": -1,
    "pingtime": 291268,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.0(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 18,
    "addr": "35.237.217.145:18333",
    "addrlocal": "114.215.41.16:41920",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 3564,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 230145,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 20,
    "addr": "67.205.179.161:18333",
    "addrlocal": "114.215.41.16:33262",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422283,
    "lastrecv": 1544422283,
    "bytessent": 1880,
    "bytesrecv": 1913,
    "conntime": 1544419163,
    "timeoffset": 1,
    "pingtime": 229860,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.0(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 34,
    "addr": "52.214.195.71:18333",
    "addrlocal": "114.215.41.16:45760",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 1880,
    "bytesrecv": 1936,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 296538,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.0(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 41,
    "addr": "217.23.14.133:18333",
    "addrlocal": "114.215.41.16:49254",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 2024,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 315166,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.17.1(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1268000,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 1,
    "addr": "118.26.129.214:18333",
    "addrlocal": "114.215.41.16:54122",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 636893,
    "bytesrecv": 99897864,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 16798,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.2(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 12,
    "addr": "139.162.89.204:18333",
    "addrlocal": "114.215.41.16:59396",
    "services": "0000000000000001",
    "relaytxes": true,
    "lastsend": 1544422313,
    "lastrecv": 1544422313,
    "bytessent": 4280,
    "bytesrecv": 4909,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 122232,
    "version": 70015,
    "subver": "/bcash:v1.0.2/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 6,
    "addr": "47.99.176.58:18333",
    "addrlocal": "114.215.41.16:40584",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5135,
    "bytesrecv": 491261,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 27974,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.2(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 50,
    "addr": "188.166.108.147:18333",
    "addrlocal": "114.215.41.16:43104",
    "services": "0000000000000036",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 2438,
    "conntime": 1544419163,
    "timeoffset": -1,
    "pingtime": 690353,
    "version": 70015,
    "subver": "/Bitcoin XT:0.11.0J(Linux; x86_64; EB32)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 46,
    "addr": "207.154.196.149:18333",
    "addrlocal": "114.215.41.16:44734",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 1968,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 333636,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.16.2(EB8.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1233077,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 63,
    "addr": "159.203.58.120:18333",
    "addrlocal": "114.215.41.16:43688",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422312,
    "lastrecv": 1544422312,
    "bytessent": 1880,
    "bytesrecv": 2329,
    "conntime": 1544419190,
    "timeoffset": 0,
    "pingtime": 237611,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.5(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1268003,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 57,
    "addr": "54.248.49.139:18333",
    "addrlocal": "114.215.41.16:40286",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422286,
    "lastrecv": 1544422286,
    "bytessent": 5257,
    "bytesrecv": 491750,
    "conntime": 1544419164,
    "timeoffset": 0,
    "pingtime": 77476,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.3(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 11,
    "addr": "54.250.148.121:18333",
    "addrlocal": "114.215.41.16:39566",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5135,
    "bytesrecv": 491256,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 93875,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.2(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 15,
    "addr": "138.68.231.122:18333",
    "addrlocal": "114.215.41.16:42562",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 2384,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 180575,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.5(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1268003,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 21,
    "addr": "165.227.127.169:18333",
    "addrlocal": "114.215.41.16:34718",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 2511,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 231902,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 67,
    "addr": "112.222.87.221:18333",
    "addrlocal": "114.215.41.16:47762",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422291,
    "lastrecv": 1544422291,
    "bytessent": 3986,
    "bytesrecv": 328982,
    "conntime": 1544419290,
    "timeoffset": 0,
    "pingtime": 58248,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.5(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273285,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 5,
    "addr": "47.99.176.58:18333",
    "addrlocal": "114.215.41.16:40536",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5135,
    "bytesrecv": 491281,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 28001,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.2(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 13,
    "addr": "3.0.44.122:18333",
    "addrlocal": "114.215.41.16:59846",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 5135,
    "bytesrecv": 491256,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 131989,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.2(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 25,
    "addr": "165.227.127.169:18333",
    "addrlocal": "114.215.41.16:34728",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422285,
    "lastrecv": 1544422285,
    "bytessent": 1880,
    "bytesrecv": 2456,
    "conntime": 1544419163,
    "timeoffset": 0,
    "pingtime": 228818,
    "version": 70015,
    "subver": "/Bitcoin SV:0.1.0(EB128.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1274400,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  },
  {
    "id": 36,
    "addr": "144.217.73.86:18333",
    "addrlocal": "114.215.41.16:48594",
    "services": "0000000000000025",
    "relaytxes": true,
    "lastsend": 1544422284,
    "lastrecv": 1544422284,
    "bytessent": 6440,
    "bytesrecv": 494602,
    "conntime": 1544419163,
    "timeoffset": 6,
    "pingtime": 261153,
    "version": 70015,
    "subver": "/Bitcoin ABC:0.18.3(EB32.0)/",
    "inbound": false,
    "addnode": false,
    "startingheight": 1273284,
    "whitelisted": false,
    "cashmagic": false,
    "bytessent_per_msg": null,
    "bytesrecv_per_msg": null
  }
]
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 14:12:10 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1241077,
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 14:12:19 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1245077,
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 14:16:38 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo | grep headers
  "headers": 1245077,
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 14:18:12 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus# /root/go/src/github.com/copernet/copernicus/tools/coperctl/coperctl getblockchaininfo
{
  "chain": "test",
  "blocks": 0,
  "headers": 1245077,
  "bestblockhash": "000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943",
  "difficulty": 1,
  "mediantime": 1296688602,
  "verificationprogress": 4.552460414806139e-8,
  "pruned": false,
  "chainwork": "0000000000000000000000000000000000000000000000000000000100010001",
  "softforks": [
    {
      "id": "bip34",
      "version": 2,
      "reject": {
        "status": false
      }
    },
    {
      "id": "bip66",
      "version": 3,
      "reject": {
        "status": false
      }
    },
    {
      "id": "bip65",
      "version": 4,
      "reject": {
        "status": false
      }
    },
    {
      "id": "csv",
      "version": 5,
      "reject": {
        "status": false
      }
    }
  ],
  "bip9_softforks": {
    "csv": {
      "status": "defined",
      "bit": 0,
      "startTime": 1456790400,
      "timeout": 1493596800,
      "since": 0
    },
    "testdummy": {
      "status": "defined",
      "bit": 28,
      "startTime": 1199145601,
      "timeout": 1230767999,
      "since": 0
    }
  }
}
root@server_16:~/go/src/github.com/copernet/copernicus# date
Mon Dec 10 14:18:56 CST 2018
root@server_16:~/go/src/github.com/copernet/copernicus#


 

mempool_reorg.py reports error

Describe the bug
mempool_reorg.py report inconsistent mempool tx

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.