Test stage fails in CI about deployment-terraform HOT 9 CLOSED

This is probably caused by the order in which validator nodes are created.

So initially there is only master of ceremony (MoC).
Then the first validator's key is added. According to AuRa [N/2]+1 nodes should seal blocks before the new N+1th node can join. In this case it's [1/2]+1 = 1 (== MoC), so the first validator is added OK.

Then the second validator's key is added, but first validator's node is not up yet. Since [2/2]+1 = 2 > 1, thus the second validator is not yet "approved", he is in "pending" state.

Our CI setup ends here (MoC + 2 validators). When the first validator's node is created, it "approves" the second validator and he can join later.

It seems that if you add the third validator, he never gets out of pending state, probably because there is already one node in pending state.

So the possible solution here is to change the ceremony playbook to create validator nodes sequentially, one after another.

from deployment-terraform.

No blocks are being added in the netstats where as I had it working in a stable manner earlier this month. I'm not sure if dependencies are broken or what...

I'm not sure what is wrong, but I made almost no changes from upstream when running it with the 2 bootnode and 3 validator setup the testing stage still fails: https://circleci.com/gh/C4Coin/deployment-terraform/198

Is this a failure in the ceremony somewhere or the initial key issuance? Will keep testing until I figure it out

from deployment-terraform.

@ArseniiPetrovich sorry to bother, but any guidance is appreciated as I've been stuck on this for a few days.

from deployment-terraform.

Hey @tavakyan, do these 3 validators include MoC? Because you have 4 validators according to test logs. From the last run they are:

0xE6DA5D442DF1b6C93Ca3501df3e171a4Ec8008F7,
0xfA2Fd6A1cA23BbF2D8eC850c24B8e47294241e05,
0xB7be5ceE0Addf14Dcb4e64ff441461aC894FEbef,
0xD98e0Be2483fd5D80f51Da94Ce558d4294E3876B

And the last one didn't mine any blocks with transactions.
The same is for previous log, 4 validators and the last one can't mine.
Can you please try to increase number of rounds? And check if all validator nodes don't have problems with connection

from deployment-terraform.

@ArseniiPetrovich sorry to bother, but any guidance is appreciated as I've been stuck on this for a few days.

Hey @tavakyan
I'm here too, joins the @natlg questions above.

from deployment-terraform.

Thank you guys. So far I realized my netstat wasn't working, but according to the block explorer blocks are being added. I changed my maxRounds to 16 since I do have 4 validators including the MoC. I still get the same test failure.

When I look through the block miners in each block one of them is definitely not mining. It seems to never be added to the list of miners at all. When I increase the validators from 3 to 4, two of them are not present... I may have had a mistake in my group_vars yml file which may be related to the # keys I generate.

See https://github.com/C4Coin/deployment-terraform/blob/alphanet/azure/group_vars/alphanet.yml

The logs look similar: https://circleci.com/gh/C4Coin/deployment-terraform/241

from deployment-terraform.

Seems network always has 3 working validators (including MoC), no matter what's in vars file. Although number of generated keys for validators matches initial_key_convert_count. Test gets this number from contract. Interesting what will happen if setup 2 or 1 validators, will it pass.

It could help to check amount of actually running nodes and their logs. Not sure how all it works with circleci and if it's possible to ssh to nodes during test.

maxRounds number doesn't greatly affect the result as one round checks number of blocks equal to number of validators. It's just to make sure they didn't miss some rounds occasionally.

from deployment-terraform.

I ran it with 2 validators and it seems to work and the tests pass. Not sure what the root cause is, but if I find out I'll open a separate issue. This configuration is fine for now. Thank you guys for the help.

from deployment-terraform.

Just a heads up. I think I may have found the issue. We were using the master branch for the poa-network-consensus-contracts which limits the bridges in the rewardbyblock contract to 3. This is probably the issue though I haven't tested it. Thanks again for all the help @ArseniiPetrovich and @natlg

from deployment-terraform.