Syncing Under Load Fails about wavelet HOT 5 CLOSED

This still fails even with #177 applied.

What I did to test it was run a benchmark against 3 nodes, then while the benchmark is running and after 30+ rounds have been pruned, restart one node.

So start a 3 node stack, 3 benchmark nodes, wait for round 60 to be reached, then do :q on the first node -- it will then try to resync, gets the error:

error adding downloaded tx to graph [transactions depth is too low compared to root:
root depth is 44288, but tx depth is 44273]: Transaction{ID: 
34d7ed5f2483b9e9457cf722110c205b0933f5b6993d7fd4f8907dc2f349eefd}

While the status looks kind of like:

8:53AM INF Here is the current status of your node. balance: 9999999999997369008 difficulty: 11 
height: 70409 id: e919a3626df31b6114ec79567726e9a31c600a5d192e871de1b862412ae8e4c0 
nonce: 25357 num_accounts_in_store: 3 num_incomplete_tx: 61223 num_missing_tx: 11
num_tx: 30653 num_tx_in_store: 91876
peers ["10.0.28.12:3001[68a5f19ea9c5e1e61836a189175a159f36dc273c07eb9e58448b3529957e3218]","10.0.28.8:3002[ce55ce8e451272e042683766f02165b8ca18ce1aa989c77f48b87ff4fc29fcfa]"] 
preferred_id: f3d0b660887f7be049d9efcdc4e317ac51fe9d411c425cd5331c957aed4ff348 
preferred_votes: 0 reward: 39148
root_id: d4f27185167851c5d4c6003f023758fc84176e0c67667befe109c6ecb487bffa
round: 61 stake: 2580280 sync_status: "Node is fully synced"

from wavelet.

First of all I'm not even sure that we should be trying cases with spamming nodes while preventing consensus from happening (having less nodes running than necessary for consensus) because that never should a case in production. @iwasaki-kenta thoughts?
But nonetheless, I tried couple of different setups and for branch with changes in vote counting setup described above is actually resolves.
I tried running 3 and 4 nodes network while benchmarking 2,3 and 3 nodes for respective setup. Restarting one of the nodes always results into target node getting back into finalization process even without stopping benchmark on any of the nodes in the network.
I should say that such test is really heavy on CPU and all potential problems might be because of nodes are not getting enough of resources to correctly run on their own.

from wavelet.

What about not updating the adding transactions or updating the graph if there are less than WAVELET_K peers ?

from wavelet.

I tested with the vote-countint-for-finalize branch and disallowing ledger.AddTransaction() (diff below) if there are less than K peers (test configuration: WAVELET_K=2, WAVELET_NODES=3, stake for each node was equal at 1000 kens)

--- a/ledger.go
+++ b/ledger.go
@@ -218,6 +218,10 @@ func NewLedger(kv store.KV, client *skademlia.Client, opts ...Option) *Ledger {
 // is returned if the transaction has already existed int he ledgers graph
 // beforehand.
 func (l *Ledger) AddTransaction(tx Transaction) error {
+       if len(l.client.ClosestPeerIDs()) < conf.GetSnowballK() {
+               return errors.New("Unable to accept transactions with less than K peers")
+       }
+
        err := l.graph.AddTransaction(tx)

        if err != nil && errors.Cause(err) != ErrAlreadyExists {

After 60 rounds I restarted node 1 and let it try to resync. These were the results:

Node 1: 1:55AM INF Here is the current status of your node. balance: 9999999999995648990 difficulty: 11 height: 168959 id: e919a3626df31b6114ec79567726e9a31c600a5d192e871de1b862412ae8e4c0 nonce: 42026 num_accounts_in_store: 3 num_incomplete_tx: 200001 num_missing_tx: 9 num_tx: 117816 num_tx_in_store: 317819 peers: ["10.0.36.5:3001[68a5f19ea9c5e1e61836a189175a159f36dc273c07eb9e58448b3529957e3218]","10.0.36.3:3002[ce55ce8e451272e042683766f02165b8ca18ce1aa989c77f48b87ff4fc29fcfa]"] preferred_id: bf9087075bde6840344fcdfcb51e09d334813cdc3bf9f4019ac260febffee15d preferred_votes: 0 reward: 64916 root_id: a9a8d5c85ddc5148f5b3879ba6281317e57386f7683d30e49e1184db289f1cf5 round: 73 stake: 4266960 sync_status: "Node is fully synced"

Node 2: 1:56AM INF Here is the current status of your node. balance: 9999999999995711420 difficulty: 11 height: 192827 id: 68a5f19ea9c5e1e61836a189175a159f36dc273c07eb9e58448b3529957e3218 nonce: 41551 num_accounts_in_store: 3 num_incomplete_tx: 0 num_missing_tx: 0 num_tx: 326823 num_tx_in_store: 389648 peers: ["10.0.36.3:3002[ce55ce8e451272e042683766f02165b8ca18ce1aa989c77f48b87ff4fc29fcfa]","10.0.36.4:3000[e919a3626df31b6114ec79567726e9a31c600a5d192e871de1b862412ae8e4c0]"] preferred_id: 403f1641a14eb25f632b4f2f738ee3afba6958beb3c57a97bed722d22751e20e preferred_votes: 0 reward: 62180 root_id: a9a8d5c85ddc5148f5b3879ba6281317e57386f7683d30e49e1184db289f1cf5 round: 73 stake: 4205480 sync_status: "Node is fully synced"

Node 3: 1:56AM INF Here is the current status of your node. balance: 9999999999995620668 difficulty: 11 height: 190315 id: ce55ce8e451272e042683766f02165b8ca18ce1aa989c77f48b87ff4fc29fcfa nonce: 42507 num_accounts_in_store: 3 num_incomplete_tx: 7581 num_missing_tx: 869 num_tx: 320853 num_tx_in_store: 391259 peers: ["10.0.36.5:3001[68a5f19ea9c5e1e61836a189175a159f36dc273c07eb9e58448b3529957e3218]","10.0.36.4:3000[e919a3626df31b6114ec79567726e9a31c600a5d192e871de1b862412ae8e4c0]"] preferred_id: 403f1641a14eb25f632b4f2f738ee3afba6958beb3c57a97bed722d22751e20e preferred_votes: 0 reward: 63260 root_id: a9a8d5c85ddc5148f5b3879ba6281317e57386f7683d30e49e1184db289f1cf5 round: 73 stake: 4294320 sync_status: "Node is fully synced"

Additionally nodes 2 and 3 were printing lots of (identical values on both nodes):

got merkle 9472adfc3d995f84a94954e27f64509c but expected 6aa8d345ab74c1fb6ad7890f22ce04fc
got merkle 9472adfc3d995f84a94954e27f64509c but expected 6aa8d345ab74c1fb6ad7890f22ce04fc
got merkle 9472adfc3d995f84a94954e27f64509c but expected 6aa8d345ab74c1fb6ad7890f22ce04fc
got merkle 9472adfc3d995f84a94954e27f64509c but expected 6aa8d345ab74c1fb6ad7890f22ce04fc
...

These seems to indicates a few things:

1. The vote-countint-for-finalize branch seems to be working, as nodes 2 and 3 ended up with the same preferred ID
2. It's not sufficient to not update the ledger when there are fewer than K peers (maybe the graph is still being updated somewhere ? Unsure, have not checked)

Also, during this time the nodes were using a significant amount of RAM (but this appears to be expected)

from wavelet.

Fixed by #243

from wavelet.