Need to cover Agent app with unit tests and add them to the pipeline

Create rich set of small Unit Tests covering all or most of all internal APIs of IMA agent.

80-90% can be covered with tests.

┆Issue is synchronized with this Jira Task

Preconditions
skaled: 3.5.7-consensus-and-asan
IMA: 1.0.0-develop.136
Schain: 1 MEDIUM

Step to reproduce
Run ima functional tests.(IMA Cats)
Observe logs

Actual result
During test execution we see decreasing performance of test after start(in test we have some await), after few iterations of sending ETH we have crash of 5 skaled:
2 of them crash 2 times in a row then continue work
2 of them stuck for 10-20 min (BlockFinalizeDownloader::downloadProposal) and then continue work
1 of them completely stuck

┆Issue is synchronized with this Jira Bug
┆Attachments: 2021-03-03 11_20_04.176028 Last snapshot creation time_ 1614770100 | 54.68.225.187.log | log0 | log1 | log2

schain IMA contracts need to

• be present in the genesis block
• be initialized
• be connected to corresponding mainnet contracts

┆Issue is synchronized with this Jira Task

IMA agent uses pending work tracker implemented as part of skaled to monitor long time mined IMA message transfers (calls to MessageProxy.postIncommingMessage() contract method where messages arrived finally). All the notifications about pending IMA messages are saved inside skaled via JSON RPC calls. Also, skaled broadcasts IMA pending work entries to other skaled instances in S-Chain via special JSON RPC broadcast calls. All the JSON RPC calls above must be ECDSA-signed.

NOTICE:

• This task is optional but good to have for test net
• It's must have for release

NOTES FOR IMPLEMENTATION:-

• The CryptoManager::sgxSignECDSA, CryptoManager::verifyECDSA, setSGXKeyInfo APIs in Consensus offer ready to use ECDSA algorithm wrappers
• The JStubClient.h file in Consensus offers ready to use configuration loader
• The setSGXKeyInfo API in Consensus offers offers ECDSA key initialization code
• The ConsensusEngine class in Consensus does ECDSA initialization from loaded config values in skaled
• The crypto/OpenSSLECDSAKey.h file is ready to use ECDSA key implementation with APIs
• The OpenSSLECDSAKey::deserializeSGXPubKey API is handy to convert string key into OpenSSL's EC_KEY that can be destroyed later with EC_KEY_free( ecKey )call

┆Issue is synchronized with this Jira Task

Preconditions
IMA: ima:1.0.0-logs.2
Skaled: schain:3.2.2-develop.0
https://skalelabs.atlassian.net/wiki/spaces/SE/pages/1173880898/Rinkeby+Testnet-281020
16 nodes schain(test type MEDIUM)
1 sgx operate all nodes

Step to reproduce
Turn off sgx for some time(15 min for example )
Send tx ETH M to S
Turn on sgx

Actual result
IMA agent stuck on all nodes
Schain name skale_ima_faint-rasalhague
IMA agent stuck: 52.38.230.168
IMA agent sing error: 54.70.3.214

┆Issue is synchronized with this Jira Bug

During schain creation IMA contracts must be registered on mainnet part

┆Issue is synchronized with this Jira Task

Preconditions
NETWORK: KOVAN
IMA: 1.0.0-develop.124
Skale manager: 1.8.0

Step to reproduce
Sent eth from M to S

Actual result
2021-01-27 16:59:41.206: M2S ETH Payment: Verifying the OutgoingMessage event of the MessageProxy/0x90E262fc3fB8C046de1a54Ebca82434589B3AB2E contract ...
2021-01-27 16:59:46.279: M2S ETH Payment: CRITICAL ERROR: Payment error in w3_main_net.eth.sendSignedTransaction(): Error: Returned error: One of the blocks specified in filter (fromBlock, toBlock or blockHash) cannot be found

One of the blocks specified in filter (fromBlock, toBlock or blockHash) cannot be found



┆Issue is synchronized with this [Jira Bug](https://skalelabs.atlassian.net/browse/SKALE-3820)

Make web3 transaction and signing execution of IMA functions wrapped into an easy to use NPM.

ima.js

index.js → 3 namespaces?

USAGE (npm if not integrated with wallet, otherwise just use wallet - metamask/portis/bitski/walletconnect, etc)

• getETH balance
• getERC20 balance
• getERC721 balance
• AddEthForExit
• Deposit ETH into SKALE Chain (token address, amount, options)
• Deposit ERC20 into SKALE Chain
• Deposit ERC721 into SKALE Chain
• Withdraw ETH from SKALE Chain
• Withdraw ERC20 from SKALE Chain
• Withdraw ERC721 from SKALE Chain

SETUP/MAPPING (mainly nodejs)

1. Deploy the ERC20 or ERC721 token contract on Mainnet.
2. Deploy the ERC20 or ERC721 token contract on the SKALE chain.
3. Grant the token’s minter role to LockAndData on the Schain.
4. LockAndDataForMainnetERC20 - addERC20TokenByOwner
5. LockAndDataForSchainERC20 - addERC20TokenByOwner

┆Issue is synchronized with this Jira Story
┆Attachments: result

We need to move from constant value and use estimateGas for ima tx.
https://rinkeby.etherscan.io/tx/0x24389a173ff59a89be26f308dd54a1c95932934b656f7649268bf7ad89572d5a

┆Issue is synchronized with this Jira Task

IMA MultiSig issue - Binance has a solution that splits the primary key (Binance threshold ECDSA). Need to investigate and see if there is anything applicable to us.

┆Issue is synchronized with this Jira Task

Disable IMA work broadcasts by default. IMA agent should not not use pending work tracking mechanism until dedicated command like key is provided.

┆Issue is synchronized with this Jira Task

Additional details and discussion background could be found in https://skalelabs.atlassian.net/wiki/spaces/SKALE/pages/1377828907/Schain+owner+should+deploy+Token

┆Issue is synchronized with this Jira Story

We still have some number of reverted transactions on MainNet/Rinkeby as result of non-unique transactions sent after one or more skaled instances were crashed or restarted. We need to find a way to minimize number of such reverted transactions as much as possible.

┆Issue is synchronized with this Jira Task

Both mainnet and schain contracts need to be paused in 2 steps:

1. suspend - all mutable functions must revert except processing messages from external chain - can be called only by schain owner
2. pause - all mutable functions must revert - can be called only by an agent (requires BLS signature verification) only if contracts are suspended

Paused contracts must be able to be unpaused by schain owner

┆Issue is synchronized with this Jira Task

Need to optimize gas usage for main IMA functions to improve the usability and user adoption

• ERC20 Transfer

┆Issue is synchronized with this Jira Task

Need to add start/stop operations to

• begin/finish accept messages on IMA
• add opportunity to schain owner to stop IMA messages
• deploy and add tokens.

[~accountid:5b2037f147afa959b14789e5] [~accountid:5b293e1e9bcd7518af2198f6] to clarify the scope

┆Issue is synchronized with this Jira Task

Add new --colors and --no-colors command line flags to turn on/off ANSI colors in log output of IMA agent.

┆Issue is synchronized with this Jira Task

IMA agent supports many log levels, The most detail log level produces too much output. The middle log level is quite reasonable and handy but in case of errors we are missing low level details which were not logged just before error occur. Accumulative logging technique means gathering high detail log info for recent events and print low level details for recent actions only when error occur.
Notice: this task is massive refactoring of all the logging in IMA agent,

┆Issue is synchronized with this Jira Task

IMA Agent must

• get version of IMA contract on mainnet and schain
• ensure that versions are compatible
• continue execution with a correct behavior corresponding to contracts version

┆Issue is synchronized with this Jira Task

Create useful and convenient tool to create correct tokens which would be suitable to use with IMA

┆Issue is synchronized with this Jira Task

Preconditions
IMA: 1.0.0-develop.148
Skaled: 3.5.12-eth-global
Geth: Berlin

Step to reproduce
Send tx transfer M >> S

node /home/evgeniy/Work/IMA/agent/main.js --gas-price-multiplier=2 --verbose=9 --m2s-payment --value=120finney --url-main-net=http://11.111.111.111:8545 --url-s-chain=http://22.222.222.22:10067 --id-main-net=Mainnet --id-s-chain=wailing-arneb --cid-main-net=4 --cid-s-chain=-4 --abi-main-net=/home/evgeniy/Work/IMA/proxy/data/proxyMainnet.json --abi-s-chain=/home/evgeniy/Work/IMA/proxy/data/precompiled.json.file --key-main-net=SOME PK --key-s-chain=SOME PK

Actual result
....signed serialized TX is <Buffer f8 f3 82 55 cb 84 3b 9a ca 00 83 02 bc 30 94 aa 5b de 88 9a e3 bb 3f 33 f3 7d c7 f3 28 a3 5e cf ba 08 6e 88 01 aa 53 5d 3d 0c 00 00 b8 84 9b e3 7a 97 ... >
2021-04-02 17:05:32.091: M2S ETH Payment: ....signed raw TX is 0xf8f38255cb843b9aca008302bc3094aa5bde889ae3bb3f33f37dc7f328a35ecfba086e8801aa535d3d0c0000b8849be37a970000000000000000000000000000000000000000000000000000000000000040000000000000000000000000d2c5b39b4e735c17612bb5a08fd024ccc5dbcb23000000000000000000000000000000000000000000000000000000000000000d7761696c696e672d61726e6562000000000000000000000000000000000000001ca0fec48d8b5b4adc97fd046defbbfd75a079e8ed00a8a059cec6e91f5d844b36d4a04db0081d8690b63cc66de5ad529dfe0783a12677b92f8ea3b0e40c9ec4b0dc6e
2021-04-02 17:05:32.161: M2S ETH Payment: WARNING: first attempt to send signed transaction failure during w3_main_net.eth.sendSignedTransaction(): Error: Returned error: only replay-protected (EIP-155) transactions allowed over RPC
2021-04-02 17:05:32.229: M2S ETH Payment: CRITICAL ERROR: second attempt to send signed transaction failure during w3_main_net.eth.sendSignedTransaction(): Error: Returned error: only replay-protected (EIP-155) transactions allowed over RPC
2021-04-02 17:05:32.229: M2S ETH Payment: CRITICAL ERROR: Payment error in w3_main_net.eth.sendSignedTransaction(): Error: Returned error: only replay-protected (EIP-155) transactions allowed over RPC

┆Issue is synchronized with this Jira Bug

The main task of IMA agent running message transfer loop. But it also supports handy command line to do ETH and token transfers. There are two token transfer options both for 20 and 721: raw token transfers and non-row token transfers. Last one auto-allocates ERC token instance on S-Chain what is removed feature now. Only raw token transfers are supported by design of latest IMA/develop. So, we need to removed non-raw token transfer commands from IMA agent and it's MD documentation to avoid customer misunderstanding.

┆Issue is synchronized with this Jira Task

Because IMA does not support schain <-> schain messages it can be simplified to reduce storage requirements and gas costs

┆Issue is synchronized with this Jira Task

Support transferring ERC-1155

┆Issue is synchronized with this Jira Story

Due to the Berlin fork, CID_MAIN_NET env variable now required for the IMA container (--cid-main-net). Default value (-4) is no longer works.

┆Issue is synchronized with this Jira Bug

Need to investigate how pausing and upgrade-ability is done in other projects, for example 0x protocol, Gnosis Safe, Moonbeam.

Basically IMA contracts should provide smart pausable interface which pauses message functionality partially. New IMA messages should be reverted and never pushed into message queue. Existing messages should be allowed for processing and delivered to target block-chain. As result, all message queues should become empty soon or late and this means IMA is finally safe to perform contracts upgrade.

┆Issue is synchronized with this Jira Task

Reimbursement transactions going to schain creator
TO DO:
Reimbursement transactions should going to validator wallet(node wallet)

• Implement Self-Recharging Wallets to MessageProxy contract on main net
• Implement whitelistOfExit on Skale-chain
• Add option to pay fee from Skale chain owner only

┆Issue is synchronized with this Jira Bug

We need a script to be executed by schain owner to perform IMA upgrade

┆Issue is synchronized with this Jira Task

SKALED does not search logs for queued IMA messages correctly what causes IMA agent to stop transfer loop. The msgCounter logs query argument must be lowercase.

┆Issue is synchronized with this Jira Bug

For both mainnet and schain contracts agent must monitor their state. If contracts are suspended and all messages are delivered it must send transaction to pause contracts.

┆Issue is synchronized with this Jira Task

Support transferring a minted schain ERC721 to mainnet
Use messageproxy

┆Issue is synchronized with this Jira Story

This Epic covers work for IMA upgradeability and upgrade flow
Spec: https://skalelabs.atlassian.net/wiki/spaces/SKALE/pages/1767768082/IMA+Upgradeability

┆Issue is synchronized with this Jira Task

Replace addresses of all built-in IMA contracts on S-Chain. New addresses should be simple-style addresses which does not correspond to any private keys. [~accountid:5b293e1f53650a265f99c7bb] already did this for built-in contracts. Make appropriate corrections of All Skale (functional test), SKALE IMA SDK, IMA Cats sample app.

┆Issue is synchronized with this Jira Task

Implement Gas Costs reimbursement logic according to the spec: https://skalelabs.atlassian.net/wiki/spaces/SKALE/pages/1713143812/Gas+cost+reimbursement

┆Issue is synchronized with this Jira Task

Token balance is stored as an integer number. To support fractional balance ERC20 has decimal parameter that shows the decimals places of the token. For example if decimal is equal to 18 and user has a balance equal to 10 ^18^ it is shown as 1.
When IMA creates clone of ERC20 on schain it always uses decimal value as 18 even if a token on mainnet has a different setting. It may cause a confusion. For example if there is a token with decimal value equal to 19 and user send 1 token to schain on schain it the balance will be 10 tokens. When user sends 10 tokens to mainnet it receives only 1 token.

┆Issue is synchronized with this Jira Bug

IMA may drain wallet if Main Net is slow (overloaded with pending transactions) and transaction mining time is equal or longer to time frame of single IMA agent. The next IMA agent will submit same transaction in this case from different wallet what means double expense of same operation. This can be fixed by pausing work of IMA agent based on pending transaction queue analysis.

┆Issue is synchronized with this Jira Task

The epic is based on the IMA gaps and additional work described here:
https://skalelabs.atlassian.net/wiki/spaces/SKALE/pages/1377828907/Schain+owner+should+deploy+Token

┆Issue is synchronized with this Jira Epic

Preconditions
Network: RINKEBY
IMA: 1.0.0-develop.124
2-nodes schain(can be reproduced on any type)
MetaMask Version
9.0.3

Step to reproduce
Create schain
Add schain in metamask:
nano .ska
cat .skale/node_data/schains/shapely-enif/schain_schain_name.json | grep "infoHttpRpcPort"
add custom network in metamtask with info port
Send eth(in my case 0.1 eth ) from mainnnet to schain

Actual result
Metamask return incorrect value on UI
IMA transfer eth tx: https://rinkeby.etherscan.io/tx/0xd80b8b9a5d100d4a859214f3b20f4420e87209ba31f37d71bbc067ee7b495410
https://rinkeby.etherscan.io/tx/0x193d8fb4786063158e41b280d5d4a077bed5d056fb032fc3dc0de82fc00c789c
geth attach to schain endpoint(infoPort )result:
instance: skaled/3.4.4+commit.f786053e/linux/gnu7.5.0/debug
coinbase: 0xa324817c935954636128693e464bd76cafb3592f
at block: 10924 (Thu Jan 28 2021 13:55:11 GMT+0200 (EET))
modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 skale:0.1 skaleDebug:1.0 skaleStats:1.0 web3:1.0

To exit, press ctrl-d

eth.getBalance("0xbc2a70Ae28f8058140B214Cea2167e9c587A5790")
200000000000000000

┆Issue is synchronized with this Jira Bug
┆Attachments: Screenshot from 2021-01-28 14-14-50.png

Latest IMA smart contract cost optimizations and cleanup introduce format changes in IMA messages. Skaled cannot verify new IMA messages. Make skaled understand new IMA message format and verify all message fields.

┆Issue is synchronized with this Jira Task

Need to register MessageProxy in ContractManager - Self-Recharging wallet requirement
Currently it blocks transfers form schain to mainnet

┆Issue is synchronized with this Jira Task

After transferring ERC721 token we need to execute depositERC721 function.
Deposit ERC721 token to Schain Flow:

1. Transfer ERC721 token to DepositBox
2. Run depositERC721

After 1 step you have lost ownership of token - so everyone can front-run 2 step

┆Issue is synchronized with this Jira Bug

IMA agent sends sequences of transactions while working. Approve and Deposit calls for instance. The code invoking these transaction sequences must be refactored to provide more correct and successful gas estimation.

┆Issue is synchronized with this Jira Task

Because openzeppelin cli is deprecated we need to use openzeppelin upgrade plugins for IMA upgrades.

┆Issue is synchronized with this Jira Task

Latest IMA/skale-manager/skaled are not fully supported by all-skale/functional test.

┆Issue is synchronized with this Jira Task

IMA message envelope validation foir ERC20 and ERC721 tokens needs to be corrected according to the latest changes in IMA smart contracts.

┆Issue is synchronized with this Jira Bug

Avoid using pending transactions scan. Replace it with own mechanism based on TX cache inside skaled. This cache should be used by all the IMA agents to understand which transactions are still pending and in-progress.

┆Issue is synchronized with this Jira Task

ImaFactory is used to deploy IMA instance on mainnet for particular chain
During the factory deployment all reference implementations of IMA contracts need to be deployed.
On new IMA instance deployment upgradable proxies pointing to reference implementations must be used.
Owner of the new instance must be schain owner.

ImaFactory needs to be registed in skale-manager after deployement.

┆Issue is synchronized with this Jira Task

Need to implement BLS signature verification for messages which comes to SKALE-chain. Currently all messages would be signed but signature checks only in S2M transfers.

• Add SkaleVerifier contract to SKALE-chain as redeployed contract
• Somehow get BLS public key(get from Skale-chain config, or add it with signature)

┆Issue is synchronized with this Jira Task

Restructure IMA contracts on Schain part following New IMA Flow

┆Issue is synchronized with this Jira Task

IMA proxy contracts contain incorrect reference to schainID.

Need to change schainID to schainName to avoid confusion.

┆Issue is synchronized with this Jira Bug