Implement ability to spin up a local test network with configured number of bakers and nodes about tezos-k8s HOT 3 CLOSED

WIP

• Currently using a statefulSet which provisions storage for each tezos node that will be spun up. Using kubectl cli to increase/decrease replicas while testing.
• Nodes are communicating with each other properly through the tezos-p2p service.
• I gradually increased the number of replicas to 32, and things started to crash at that point. Need to look more into resource provisioning for all of the nodes.

Occasionally I am experiencing issues with logs looking similar to:

Sep  4 20:47:59 - validator.peer: Worker started for NetXHFw7TkU5h:idsB7s3QQeVB
Sep  4 20:47:59 - block_validator_process_external: The process terminated abnormally with value 1
Sep  4 20:47:59 - block_validator_process_external: Block validator started on pid 639
Sep  4 20:47:59 - validator.block: Validation of block BM1nDiexDsYU7iF4YVXMsCU5jN3bSX2exvj3VUGfPqHHmdef3AR failed
Sep  4 20:47:59 - validator.block: Request pushed on 2020-09-04T20:47:59-00:00, treated in 3.459us, completed in 69.801ms ,
Sep  4 20:47:59 - validator.block: (Invalid_argument "Json_encoding.construct: consequence of bad union")
Sep  4 20:47:59 - node.validator.bootstrap_pipeline: unexpected error while fetching headers
Sep  4 20:47:59 - validator.peer: Worker crashed [node_peer_validator]:
Sep  4 20:47:59 - validator.peer: (Invalid_argument "Json_encoding.construct: consequence of bad union")

This may be related to similar issues I'm sometimes seeing with the activate-job #18.

from tezos-k8s.

@harryttd with the new approach on #29 , I believe we can have all nodes (progenitor and regular nodes) come online at the same time, since they now have all the data that they need in a configmap.

As a consequence, it would no longer be necessary to run kubectl commands from within the k8s environment.

The feature can be made part of mkchain - just specify a desired number of nodes as a parameter.

from tezos-k8s.

Closing as duplicate of the #35

from tezos-k8s.