Please add following checks to Telegram Notifications for Validator notifications

-SSL Certifications 
-Node Monitoring
-Node Balances

Preconditions
Active MEDIUM schain

Versions:
SKALE_NODE_CLI_VERSION=2.0.0-develop.24

Steps to reproduce:

1. Run

skale node turn-off

1. Wait until node was successfully turned-off
2. Run

skale node turn-on --sync-schains .env

Actual state:
Error node-on during skale_nginx creation stage
The next variables are present in the .env file and not implemented to skaled configs:

• DEFAULT_GAS_LIMIT
• MAX_GAS_PRICE_WEI
• DEFAULT_TIMEOUT
• FILESTORAGE_MAPPING
• SCHAINS_MNT_DIR

┆Issue is synchronized with this Jira Bug

Versions

• Node-cli 1.1.1-beta.1
• CONTAINER_CONFIGS_STREAM=1.7.3-testnet

Steps to reproduce

1. Create 2 schains
2. Remove 2 schains
3. Create new 2 schains
4. Run cmd

sudo skale schains dkg

Expected state:
Cleaner should remove deleted sChains from "sChains dkg" list

┆Issue is synchronized with this Jira Bug

Set up code coverage pipeline (unit testing)

• skale-admin
• node-cli

┆Issue is synchronized with this Jira Task

When a user creates a chain using CLI, verify that ChainID is 31 bit.

If not, abort.

With proper timing upgrade procedure may start to remove image during container creation. We should avoid this.

1. Add to skale-admin check for running sChain processes (should also appear in health API in watchdog/supervisor)
2. Add an API to request only selected checks
3. Update node-cli
4. Remove RPC check!
5. ???

┆Issue is synchronized with this Jira Task

Currently docker-lvmpy is running using default python version for 18.04 node, which is 3.6 and it has been deprecated for a long time. We need to change installation procedure in a way that newer version of python (such as 3.11) is used.

Description: Multiple validators would like to change the commission fees because they have new customers and new commission fees set. Some validators have a lot of nodes, and these nodes are supporting skale chains, so no Exit is possible here to reregister with a new validator.

Main Solution: Ability to change commission fee.

Alternative : Be able to move nodes from one validator to another by link/unlink node.
It would be great at least if they could move a node from one validator to another to change the commission fee.

In the current version, sChain container pulls just before the start by admin. Instead, it's better to download it in advance during init and update procedures. This should help with Cretated state issue.

┆Issue is synchronized with this Jira Task

.skale folder now stored in the home directory which may lead to errors in cases when the node was inited from one user and then some commands were executed from another.

Possible solution: save user during skale node init and then check it in all commands.

┆Issue is synchronized with this Jira Task

install validator & node cli on a new machine.
copy ~/.skale/node_data/ from the old machine to the new one
run init on the new machine.

$ skale node info
This SKALE node is not registered on SKALE Manager yet

PS: sgx wallet is on a separate machine in the LAN. on the old machine, skale wallet info brought up 0x2a2f368c8e6d269d3f87bda4e9d1d10640f7e8b7. on the new one, it brings 0x08f8ed68b60da3d1db08645f920d5cf57c759233. I don't see how a node can use a previous registration if sgx-wallet returns a different address.

Preconditions

Active schain Medium type

Versions
skale_api: skalenetwork/admin:2.2.0-develop.17

Steps to reproduce:

1. Run cmd skale exit status
2. check skale_api container logs

Actual state: Error in skale_api container logs after call node exit status

Expected state: Skale_api should return a readable response on the node exit status cmd

┆Issue is synchronized with this Jira Bug
┆Attachments: skale_api_exit_status_error.txt

Currently, resource allocation files generated dynamically which causes different allocation files on different machines.

This should be fixed:

• Set default disk sizes for Testnet/QA/Mainnet
• Check that real disk size is equal or bigger than the requirements for the current env.

┆Issue is synchronized with this Jira Task

SKALE node CLI operations moved from Bash to Python. Now tests should be updated to check Python logic.

┆Issue is synchronized with this Jira Task

After we edit some files in .skale/config and run skale node update, directory config is deleted
Node-cli: 2.0.1-beta.2

Steps:
1)Add random symbols in filebeat.yml and containers.json
2) Run skale node update
3) After failed or successful update check .skale for config directory

┆Issue is synchronized with this Jira Bug

after update node_cli version to 1.1.0-beta.10 SGX_SSL_KEY_PATH was not set after schain creation

version:
node_cli: 1.1.0-beta.10

from docker inspect <ima>:

        "Config": {
            "Hostname": "ip-172-31-12-17",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "SCHAIN_DIR=/skale_node_data/schains/buqubnvtFM",
                "MAINNET_PROXY_PATH=/skale_vol/contracts_info/ima.json",
                "SCHAIN_PROXY_PATH=/skale_node_data/schains/buqubnvtFM/proxy.json",
                "SCHAIN_NAME=buqubnvtFM",
                "SCHAIN_RPC_URL=http://127.0.0.1:10131",
                "MAINNET_RPC_URL=http://15.188.54.155:8545",
                "NODE_NUMBER=13",
                "NODES_COUNT=16",
                "SGX_URL=https://45.63.65.79:1026",
                "ECDSA_KEY_NAME=NEK:76ee90f77ea01bb501bea180e904cc66d301099383ec53ffcb5f53d7e1e710ce",
                "SGX_SSL_KEY_PATH",
                "SGX_SSL_CERT_PATH",
                "NODE_ADDRESS=0x33613b84eb80d644013be60d90f1e112573ca58c",
                "TM_URL_MAIN_NET=http://127.0.0.1:3008",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NODE_VERSION=10.15.1",
                "YARN_VERSION=1.13.0"
            ],

┆Issue is synchronized with this Jira Bug

Allow validator to register a domain name in node-cli

Existing validators should be able to update the domain name

┆Issue is synchronized with this Jira Task

Admin and node-cli should wait for 1 minute after stoping skaled before removing it. Currently, it's 40 seconds.

┆Issue is synchronized with this Jira Task

Currently skale logs dump command downloads an archive with logs only from docker containers. The suggestion is to add node-cli and .removed_containers logs to the archive.

┆Issue is synchronized with this Jira Task

For sync node CLI we should be able to enable/disable block rotation during node init/update:

• .env file or cli option could be used
• then the option should be saved somewhere and passed to sChain config

After instance restart we get "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" error
Steps:

1. reboot instance
2. wait for instance start
3. docker ps

┆Issue is synchronized with this Jira Bug

If a validator back up and restores in a new machine with a different mount_point path restore operation fails.

Because DISK_MOUNTPOINT_FILEPATH reading from the back up file (which is SKALE_DIR, 'node_data' file), not from the newly defined by the validator [ENV_FILE]

skale node restore [BACKUP_PATH] [ENV_FILE]

┆Issue is synchronized with this Jira Task

--gas-price option should be added in the set-domain

┆Issue is synchronized with this Jira Bug

Currently, internal sChain volume limits are calculated in 2 different ways for the different pieces. Consensus, LevelDB, and filestorage limits are calculated dynamically, but the storage limit is pre-set.

Everything should be generated in advance using static file in skale-node (configs.yml) and Python script in helper-scripts. It should generate schain_allocation.yml file with all params (in bytes)

┆Issue is synchronized with this Jira Task

Change medium schain configuration to 1/8 of a node instead of 1/32

Update resource allocation and scratch space

┆Issue is synchronized with this Jira Task

STR

1. create backup node A on node_cli v.1
2. restore on some node B from node A
3. update node_cli from v.1 to v.2 for node B
4. run backup node B

actual: have an issue with mysql
expected: backup.tar.gz success created

node_ip: 54.245.197.145

┆Issue is synchronized with this Jira Bug

This file is reduntant and not used anymore

SGX_SERVER_URL_FILENAME = 'sgx_server_url.txt'

┆Issue is synchronized with this Jira Task

Currently, those settings are stored in the schain_static params file and not depend on the env type. Should be moved to configs.yml with different values for different env types.

┆Issue is synchronized with this Jira Task

Node id is important for troubleshooting issues and right now it's impossible to get it without manually opening node_config.json file. We need to add this data into skale node info command output.

┆Issue is synchronized with this Jira Task

Currently, we have only 2 main commands in sync-node-cli:

• sync-node init
• sync-node update

It would be useful to have turn-on and turn-off commands that works in the same way as for the regular node-cli.

skale logs dump [PATH]

This command now works via skale-api docker container.
Since now we have all required logic in the node-cli, logs dump command should be implemented in the node-cli only.

┆Issue is synchronized with this Jira Task

skale schains ls command doesn't work as expected in node-cli v2.
It should return all sChains assigned to the node on smart contracts, instead, it returns sChains on the node.

┆Issue is synchronized with this Jira Bug

This issue might be better off filed in the skale-manager repo but I'm filing it here because I encountered it while using the skale cli tool.

background

I am setting up a 2nd node for my validator.

config:

SGX_SERVER_URL=https://<redacted>:1026/

# use your 200gb external storage mount point
# this was manually created and attached using azure web gui
DISK_MOUNTPOINT=/dev/sdc

# This will be used for later testing.
# It’s for interchain messaging agent feature for dApp developers.
# For now you don’t have to worry about this part. We will provide more information later on.
IMA_CONTRACTS_ABI_URL=https://skale-contracts.nyc3.digitaloceanspaces.com/beta/ima.json

# This manager contracts abi url is where the SKALE manager smart contract ABI lives.
# This can change based on the beta versions or develop versions...
# see: https://forum.skale.network/t/examples-environment-variables-hard-upgrade/197
MANAGER_CONTRACTS_ABI_URL=https://skale-contracts.nyc3.digitaloceanspaces.com/fuji-testnet/manager.json

# "This is for elastic search and for skale to monitor during testnet.
# It will be retired for the mainnet.
# It would be very cool if you can add this variable for now."
FILEBEAT_HOST=3.17.12.121:5000

CONTAINER_CONFIGS_STREAM=alpine-hard-upgrade
DOCKER_LVMPY_STREAM=beta
DB_PORT=3306
DB_ROOT_PASSWORD=test
DB_PASSWORD=test
DB_USER=test
IMA_ENDPOINT=rinkeby.infura.io/v3/<redacted>
ENDPOINT=https://rinkeby.infura.io/v3/<redacted>
NODE_CLI_SPACE=develop
SKALE_NODE_CLI_VERSION=0.10.0-develop.2

# see also: https://forum.skale.network/t/example-environment-variables/106


# Variables added during hard upgrade:

# True/False will enable monitoring containers (filebeat, cadvisor, prometheus)
# Required for TestNets
MONITORING_CONTAINERS=True

steps:

# initialize skale node
skale node init --install-deps ./env

# initialize skale validator
sk-val init \
  -e wss://rinkeby.infura.io/ws/v3/<redacted> \
  -c https://skale-contracts.nyc3.digitaloceanspaces.com/fuji-testnet/manager.json \
  --wallet software

# get your skale validator signature
export VALIDATOR_NAME=staked
export VALIDATOR_ID="$(sk-val validator ls | grep $VALIDATOR_NAME | awk '{ print $2 }')"
export SIGNATURE="$(skale node signature $VALIDATOR_ID | cut -d' ' -f2)"

# link your validator address to your node address
# (NOTE: assumes you have the validator private key saved to validator-pk.txt)

export NODE_ADDRESS="$(skale wallet info | grep Address | cut -d' ' -f2)"
sk-val validator link-address $NODE_ADDRESS $SIGNATURE --yes --pk-file ./validator-pk.txt
# ✔ Node address 0x9e6737c6C3B... linked to your validator address

export NODE_NAME='skale-node-vm'   # new node name
export IP_ADDRESS="$(curl ifconfig.co)"
export PORT_RANGE_START='10000'

skale node register --name $NODE_NAME --ip $IP_ADDRESS --port $PORT_RANGE_START

tx failure

The link-address transaction from my validator succeeds, but the register-node transaction from my node fails.

logs

Here are the logs from my skale_api container:

[2020-08-26 01:15:37 +0000] [8] [DEBUG] POST /create-node
[2020-08-26 01:16:08,096 INFO] core.node - MainThread -
====================================================================================================
Node create started
IP: 52.249.250.53
Public IP: 52.249.250.53
Port: 10000
Name: skale-node-secret-vm
====================================================================================================

[2020-08-26 01:16:08,336 INFO] skale.contracts.manager - MainThread - create_node: 52.249.250.53:10000, public ip: 52.249.250.53 name: skale-node-secret-vm
[2020-08-26 01:16:08,342 INFO] skale.transactions.tools - MainThread - Dry run tx: createNode, sender: 0x9e6737c6C3B69397c439d1D2084D8d1dDB8669C9, wallet: RPCWallet, gasLimit: 3500000
[2020-08-26 01:16:08,383 INFO] skale.transactions.tools - MainThread - Tx: createNode, sender: 0x9e6737c6C3B69397c439d1D2084D8d1dDB8669C9, wallet: RPCWallet, gasLimit: 3500000, gasPrice: 1250000000
[2020-08-26 01:16:51,156 INFO] skale.utils.web3_utils - MainThread - Current block number is 7083252, waiting for 5 confimration blocks to be mined
[2020-08-26 01:18:26,742 ERROR] core.node - MainThread - Node creation failed
Traceback (most recent call last):
  File "/usr/src/admin/core/node.py", line 91, in register
    confirmation_blocks=NODE_REGISTER_CONFIRMATION_BLOCKS
  File "/usr/local/lib/python3.7/dist-packages/skale/contracts/base_contract.py", line 84, in wrapper
    tx_res.raise_for_status()
  File "/usr/local/lib/python3.7/dist-packages/skale/transactions/result.py", line 98, in raise_for_status
    raise TransactionFailedError(f'Transaction failed. '
skale.transactions.result.TransactionFailedError: Transaction failed. See receipt AttributeDict({'blockHash': HexBytes('0x712fd83da4980afc80cc99fe4426db13f73c2b8540dc3e885edcf284ced62507'), 'blockNumber': 7083252, 'contractAddress': None, 'cumulativeGasUsed': 7291385, 'from': '0x9e6737c6C3B69397c439d1D2084D8d1dDB8669C9', 'gasUsed': 3359503, 'logs': [], 'logsBloom': HexBytes('0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'), 'status': 0, 'to': '0x139360fc743b1DA2adb43F5999E28998Da3b6285', 'transactionHash': HexBytes('0xf6d5a00065681a0b0f1bf5fe8068e881db2fd589d2ecfbcfc5014038a929bbbf'), 'transactionIndex': 1})

Now we are generating nginx config in .skale/configs , but should generate it in .skale/node_data dir

┆Issue is synchronized with this Jira Task

Description: Nodes that do not have a stake and support the SKALE chain should be able to unregister.

Leading Solution: If a delegator undelegates, nodes supporting SKALE chains should have the ability to exit, and chains should be assigned to new validator nodes. Use node rotation to assign the SKALE chain to the new validator node.

Alternative solution:
Often, delegators want to change a different commission fee or a different validator. Just like we give the option to change the validator's public address to something else... Nodes can be reassigned to another validator.

skale-node-cli currently has no test for commands. So I think we should set them up in another task/patch

Originally posted by @badrogger in #14 (comment)

it would be VERY helpful if the component managing iptables used /etc/services to look for which port to keep open for SSH, instead of assuming 22. we don't use standard SSH ports and thus we have to resort to cron every 5 minutes making sure that the alternate port is open.

We audit using lynis: CISOfy/lynis#604 (comment)

┆Issue is synchronized with this Jira Task

Command skale exit status shows incorrect result output if the node is in Maintenance mode

Preconditions
The node is in Maintenance mode

To Reproduce
Run commands:
skale exit start
skale exit status

Expected behavior
Output should be like "Node exiting is in process"

Actual result
Output is "Node exiting is completed"

Versions
node-cli: 2.1.0

IMPORTANT!!!
skale exit start shouldn't start AT ALL if the node is in Maintenance mode and the proper message should be shown

Describe the bug
If anything present stderr in docker-compose -v output compliance check will be failed. It can corrupt init and upgrade procedures for validators. We need to make sure we parsing only stdout output.

To Reproduce
Steps to reproduce the behavior:

Download node-cli on Ubuntu 18.04 machine with AWS
Run skale node init
Expected behavior
docker-compose check should return ok

Update test cli according to this changes:
https://github.com/skalenetwork/skale-manager/pull/493/files
We need to have opportunity to specify schain owner during schains creations in test

┆Issue is synchronized with this Jira Task

skale version returns only node-cli version. We need to see if the node is upgraded or in the previous version. Can we add better versioning to the skale version command?

AWS nodes was running from ubuntu user

STR:

1. update node_cli to 2.0.0-develop.19
2. run 'sudo skale node update'

expect: user in file /etc/skale/config.json should be ubuntu
actual: user in file /etc/skale/config.json is root

┆Issue is synchronized with this Jira Bug

When we're stopping and removing skaled containers in the node-cli, logs are saved before the stop signal.
Instead of this, we should save logs after the stop signal and before removing the container.

┆Issue is synchronized with this Jira Bug
┆Attachments: skale_schain_quiet-pherkad-minor-2.log

Modern CLI collects anonymous data from users containing information like what kind of commands users are running on their machine and demographics. It does not have any private data. Telemetry is an essential feature for any CLI. You can work on what you can measure.

We can have the following functionalities:

• Enable Telemetry support in the CLI
• Push the telemetry data on Google Analytics to give you a real-time overview of CLI usage.
  Please refer to this doc: https://cloud.google.com/service-infrastructure/docs/telemetry-reporting
• Consume the data from GA to Grafana, and produce meaningful graphs.

Since docker may not clean unused inodes correctly, we need to execute
the following command at the end of each skale container update.

docker system prune -a

┆Issue is synchronized with this Jira Task

When I run skale ssl upload and it fails due to permission problems I'd expect to get a non-zero exit code. This would make automation far easier because I wouldn't need to parse the output to determine if the command was successful.

$ skale ssl upload -c fullchain.pem -k privkey.pem
Command execution failed with [Errno 13] Permission denied: 'privkey.pem'. Recheck your inputs
Traceback (most recent call last):
  File "main.py", line 111, in <module>
  File "site-packages/click/core.py", line 829, in __call__
  File "site-packages/click/core.py", line 782, in main
  File "site-packages/click/core.py", line 1259, in invoke
  File "site-packages/click/core.py", line 1259, in invoke
  File "site-packages/click/core.py", line 1066, in invoke
  File "site-packages/click/core.py", line 610, in invoke
  File "cli/ssl.py", line 71, in upload
  File "core/helper.py", line 188, in upload_certs
PermissionError: [Errno 13] Permission denied: 'privkey.pem'
2021-03-09 17:00:22,278 - [Errno 13] Permission denied: 'privkey.pem'

Followed by:

$ echo $?
0

Would be great if this could return something other than 0.

┆Issue is synchronized with this Jira Task

Version: 2.0.1-develop.25
Precond: few schains created by PK and multisig on network
Steps:
Call skale schains ls
Call skale schains show-rules name
Call skale health schains
Call skale node check
Expected: correct responses
Actual: see logs

┆Issue is synchronized with this Jira Bug

Versions

DOCKER_LVMPY_STREAM=1.0.1-develop.6
node-cli 2.0.1-develop.27

skaled RotationHistoryOlehTest6 - custom build

Steps to reproduce

1. run skale node update with versions without checkouting branches
2. check skaled containers - docker ps -a
3. manually attach volume: mount /dev/mapper/schain_name /mnt/schain_name
4. restart docker container

Actual state: Error response from daemon: Cannot restart container skale_schain_harsh-gorgonea-secunda: error while mounting volume '': VolumeDriver.Mount: {"Err": 2}
Expected state: Skaled volume should be attached automatedly without errors

Note: Manually restart skaled after the error from the actual state will repair skaled container

┆Issue is synchronized with this Jira Bug

IMA container should be optional. IMA should be enabled by default.

┆Issue is synchronized with this Jira Task