Problem

The following gRPC/API endpoints fail:

  rpc OutgoingTxBatches(QueryOutgoingTxBatchesRequest) returns (QueryOutgoingTxBatchesResponse) {
    option (google.api.http).get = "/gravity/v1beta/batch/outgoingtx";
  }
  rpc OutgoingLogicCalls(QueryOutgoingLogicCallsRequest) returns (QueryOutgoingLogicCallsResponse) {
    option (google.api.http).get = "/gravity/v1beta/batch/outgoinglogic";
  }

They fail because they clobber with the following:

  rpc LastPendingBatchRequestByAddr(QueryLastPendingBatchRequestByAddrRequest) returns (QueryLastPendingBatchRequestByAddrResponse) {
    option (google.api.http).get = "/gravity/v1beta/batch/{address}";
  }

Example:

https://api.blue.main.network.umee.cc/gravity/v1beta/batch/outgoingtx

This should not return any error, but it is because it's being matched and handled by LastPendingBatchRequestByAddr, i.e. it's treating outgoingtx as a value for address.

Solutions

1. Either fix the ordering (I haven't tested if this actually fixes it)
2. Change "/gravity/v1beta/batch/{address}" -> `"/gravity/v1beta/batch" where it takes a query parameter (the correct way).

(2) would be the ideal way IMO. In general, it's best to avoid URL path params when possible.

seems like there is no actions implemented when the wrong destination address is specified

log:

[2021-12-30T14:24:11Z WARN  gravity_utils::types::ethereum_events] Event nonce 42 sends tokens to 0x00000000000000000000000089bde264cc4e819326482e041d4ae167981935ce which is invalid bech32, these funds will be allocated to the community pool
[2021-12-30T14:24:11Z INFO  orchestrator::ethereum_event_watcher] Oracle observed deposit with sender 0x5F74a2db08D717c94457c550af54548C4241Ace9, destination None, amount 9999999999999, and event nonce 42
[2021-12-30T14:24:12Z ERROR orchestrator::main_loop] Failed to get events for block range, Check your Eth node and Cosmos gRPC CosmosGrpcError(RequestError { error: Status { code: InvalidArgument, message: "cosmos receiver: decoding bech32 failed: invalid index of 1: invalid request", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None } })
InvalidLength

This is an expansive meta-issue covering the challenge of adding multiple EVM chain support to Gravity Bridge.

• Scope required nonces, signatures , and store data behind a chain name
• Genesis save/load scoped by chain name
• Add endpoint to list all registered chains
• Modify endpoints to be scoped by chain name
• Create governance proposal to add a new chain to the system
• Modify the orchestrator to use the new endpoints
• Add to the orchestrator configuration to have multiple sets of EVM rpc servers

This list provides a broad overview of what needs to be done. The Gravity.sol contract will remain totally untouched as it requires no modifictaions.

The Gravity module will be moved to scope all signatures, nonces, and other data behind a chain name, selected by consensus vote. Updating the store for this is the first step.

After that we update the endpoints and get the orchestrator working with them.

Some things, like signatures can be mixed into a single feed and require very little modification. The orchestrator should submit a BSC and ETH signature at the same time without issue.

Other things like the oracle flow and relayer will probably just be scoped, with one duplicate thread running for each chain.

Finally we add the governance proposal to execute a chain addition and make sure it's tested in the integration tests.

At every stage during the development here we should get the chain back to a working state and be able to roll out an update, this won't be a long running branch but a series of incremental steps.

The one thing I would like to keep in mind is that while we can make it extremely easy to add a new EVM chain to the system from a technical level the increased burden on the validators of each full node they are required to run is non-trivial. This is why I don't suggest worrying too deeply about optimizing for a million EVM chains, think on the other of a dozen being the most the validator community will probably support.

gbt requires Cosmos phrase and Ethereum private key to be provided through command line when starting the Orchestrator, or they will be read from ~/.gbt/keys.json file. In both cases they are exposed as a plain text.

This integration with the feegrant module will allow the validators to continue to submit the required claims and signatures to keep the bridge moving without paying fees.

Right now this isn't an issue because fees are not enabled at all on Gravity Bridge, but it will become one once fees are enabled.

This ticket is around validation of the ERC20 metadata proposal and ways we can make it less error prone

1. in the cli when going to submit the proposal we should make sure that the denom exists
   https://github.com/Gravity-Bridge/Gravity-Bridge/blob/main/module/x/gravity/client/cli/tx.go#L49

2. make sure the token that we are setting metadata for has some amount in the total bank supply, if none exists fail the submission

https://github.com/Gravity-Bridge/Gravity-Bridge/blob/main/module/x/gravity/keeper/governance_proposals.go#L214

With the addition of this commit c440b09 we can now run invariants in unit tests. It seems like we have several issues to catch with this.

In types/key.go we sometimes change the indexing format without fully updating all prefix iterators across the codebase. This of course is not a great thing to do.

In order to reduce the risk of doing this again we should provide functions like this. In this example the prefix key is always constructed using a function, so no iterator would accidentally construct the wrong key.

See this for motivating issue ecc033d

func GetValsetConfirmNoncePrefix(nonce uint64) {
       return ValsetConfirmKey + ConvertByteArrToString(UInt64Bytes(nonce))
}

func GetValsetConfirmKey(nonce uint64, validator sdk.AccAddress) string {
	if err := sdk.VerifyAddressFormat(validator); err != nil {
		panic(sdkerrors.Wrap(err, "invalid validator address"))
	}
	return GetValsetConfirmNoncePrefix(nonce) + string(validator.Bytes())
}

I now have Gravity bridge running on Mac M1 and a Fedora box (digital droplet).

If I run any of the tests in ./all-up-test.sh, once it gets to the Rust compiler phase it takes almost an hour to compile all of the Rust files on Mac, however it seems to take only around 10 minutes on Fedora.

Spoke to @jtremback about this, just wanted to confirm there is in fact a big performance difference between the two platforms.

Currently we wait only our usual timeout amount of time to see if a batch or valset update has been submitted to Ethereum.

This isn't really a great idea as we may submit a duplicate if the relayer loop runs faster than our tx enters a block, resulting in wasted gas.

Instead we should wait up to 10 minutes for a tx we have submitted to enter the chain.

This is as simple as setting a 10 minute duration here

https://github.com/Gravity-Bridge/Gravity-Bridge/blob/main/orchestrator/relayer/src/valset_relaying.rs#L99

and here

https://github.com/Gravity-Bridge/Gravity-Bridge/blob/main/orchestrator/relayer/src/batch_relaying.rs#L306

Eitehr passing hte timeout down the call stack or declaring it there as a constant.

The docs say:

gravity keys add <my validator key name> --recover <your seed phrase>

but when you run it with --recover you get this:

$ gravity keys add timeforatest --recover seedphrase 
Error: accepts 1 arg(s), received 2

Private keys for Deep Space and Clarity should implement the zeroize-on-drop trait.

In abci.go->pruneValsets(), when removing validator sets that are no longer needed in the store, confirmations for those validator sets are not removed. The same happens with the batches and their confirmations in batch.go->OutgoingTxBatchExecuted().

This is due to a recent change in fa6f638.

When a validator completely unbonds, it gets automatically jailed. This happens in the slashing procedure of x/staking, which runs before x/gravity's slashing. So when we arrive at this portion of the code, the validator is already jailed and no slashing is executed.

Also no test is properly testing this, I'm currently looking into it but I don't know enough about it to solve it; would love to get some eyes here.

EDIT: the consensus power of a not bonded validator is always 0. So even if this is in place, the validator never gets slashed

Hey. I've been reading about bridges that connect Cosmos and Ethereum ecosystem (there are quite a few of them) and finally arrived at this repo. Having read https://blog.cosmos.network/the-gravity-bridge-chain-is-coming-to-cosmos-7010ec7bd0ea my understanding is that Gravity Bridge is going to be deployed as an independent blockchain that will use Cosmos Hub's validators via the Interchain Security. Does that sound correct? If that's the case - what's the current roadmap and when can we expect Gravity Bridge be fully deployed and usable via IBC calls and Cosmos Hub?

On the current testnet (gravity-bridge-test{1..4}), my validator is not able to withdraw rewards or commission, but is able to successfully send other transactions like transferring tokens.

withdrawing:

./gravity tx distribution withdraw-rewards gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9 --commission --from validator --keyring-backend pass
{"body":{"messages":[{"@type":"/cosmos.distribution.v1beta1.MsgWithdrawDelegatorReward","delegator_address":"gravity156n6x0hvfd2zxq5vhehp2e20mz6tdskhxtlf33","validator_address":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9"},{"@type":"/cosmos.distribution.v1beta1.MsgWithdrawValidatorCommission","validator_address":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9"}],"memo":"","timeout_height":"0","extension_options":[],"non_critical_extension_options":[]},"auth_info":{"signer_infos":[],"fee":{"amount":[],"gas_limit":"200000","payer":"","granter":""}},"signatures":[]}

confirm transaction before signing and broadcasting [y/N]: y
{"height":"29485","txhash":"A612BAF0FB4B6A46D4DF51ADE93C746BF35366408A75A5CA2E0AEB2653F75180","codespace":"undefined","code":111222,"data":"","raw_log":"panic message redacted to hide potentially sensitive system info: panic","logs":[],"info":"","gas_wanted":"200000","gas_used":"97657","tx":null,"timestamp":""}

transferring:

./gravity tx bank send validator gravity15lpnf70ycjxyhhjysmlkm6k4uf0sm220ftqy79  1000000ugraviton --keyring-backend pass
{"body":{"messages":[{"@type":"/cosmos.bank.v1beta1.MsgSend","from_address":"gravity156n6x0hvfd2zxq5vhehp2e20mz6tdskhxtlf33","to_address":"gravity15lpnf70ycjxyhhjysmlkm6k4uf0sm220ftqy79","amount":[{"denom":"ugraviton","amount":"1000000"}]}],"memo":"","timeout_height":"0","extension_options":[],"non_critical_extension_options":[]},"auth_info":{"signer_infos":[],"fee":{"amount":[],"gas_limit":"200000","payer":"","granter":""}},"signatures":[]}

confirm transaction before signing and broadcasting [y/N]: y
{"height":"14131","txhash":"64FD8BDD953639CA0CB15CFBCF6A61B3028279F4A552ABBC075221F0E0D68563","codespace":"","code":0,"data":"0A1E0A1C2F636F736D6F732E62616E6B2E763162657461312E4D736753656E64"...

I was able to change my commission and withdraw once (maybe during gravity-bridge-test1 or gravity-bridge-test2), but something happened and I'm not able to anymore.

I am able to withdraw as a different delegator to my validator:

./gravity tx distribution withdraw-rewards gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9 --from mine --keyring-backend pass
Default sign-mode 'direct' not supported by Ledger, using sign-mode 'amino-json'.
{"body":{"messages":[{"@type":"/cosmos.distribution.v1beta1.MsgWithdrawDelegatorReward","delegator_address":"gravity1pqk05hdkyhlvj07tept96cwa3fj7axasq8cg0h","validator_address":"gravityvaloper156n6x0hvfd2zxq5vhehp2e20mz6tdskhhqxhm9"}],"memo":"","timeout_height":"0","extension_options":[],"non_critical_extension_options":[]},"auth_info":{"signer_infos":[],"fee":{"amount":[],"gas_limit":"200000","payer":"","granter":""}},"signatures":[]}

confirm transaction before signing and broadcasting [y/N]: y
{"height":"35806","txhash":"65A5654C7D0F79AE6FB67DBE4D8F0B5226E71A81BFBB3C7A972AA50EE65E6381","codespace":"","code":0,"data":"0A390A372F636F736D6F732E646973747269627574696F6E2E763162657461312E4D7367576974686472617744656C656761746F72526577617264",

They are currently being checked as strings, meaning that if you register an eth address that's not checksummed (with the upper case and lower case mix), you won't be able to send any claims.

https://github.com/Gravity-Bridge/Gravity-Bridge/blob/v1.4.2/module/x/gravity/keeper/msg_server.go#L342

If you try sending a non-checksummed one in the msg, then the signature check will fail here because it always returns the checksummed one:

https://github.com/Gravity-Bridge/Gravity-Bridge/blob/v1.4.2/module/x/gravity/types/ethereum_signer.go#L63

Suggestion: check them as byte slices instead of strings.

Gravity currently isn't using the upgrade module, there are a couple guides to using the upgrade module so we need to be certain we are following all the directions out there:

https://docs.cosmos.network/master/ibc/upgrades/quick-guide.html
https://docs.cosmos.network/master/migrations/chain-upgrade-guide-044.html#chain-upgrade

Note: These two links may not be the full list of instructions to consider.

not entirely sure what's happening here, but it causes CI runs to never finish correctly.

This conditional should detect if ALCHEMY_ID is empty and if so simply pass the test

https://github.com/Gravity-Bridge/Gravity-Bridge/blob/main/tests/container-scripts/run-testnet.sh#L63

This is causing every outside contributors integration tests to show red for no good reason. Once we approve the pull the final run will execute correctly, but not before.

If a user/app wants to use the gravity bridge (module) as a dependency, they currently have to rely on a git hash. e.g. https://github.com/umee-network/umee/pull/369/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6

I believe the top-level/root package needs to include a go.mod and that go.mod needs to include the sub-module. e.g. https://github.com/PeggyJV/gravity-bridge/blob/main/go.mod

This way, users can import the module via the semantic version 👍

Because validators are expected to validate for a block or two after they're no longer bonded (Cosmos has been looking at this issue for a while) there is a case that causes certain simulations to panic. This happens in EndBlock->createValsets->GetCurrentValset

We ran into this issue here when trying to implement multi-seed sims. Maybe a check should be added to see if validators are empty and return an error that's then logged instead of panicing? e.g.

validators := k.StakingKeeper.GetBondedValidatorsByPower(ctx)
if len(validators) == 0 {
   return types.Valset{}, fmt.Errorf("validator set is empty")
}

ctx.EventManager().EmitEvent() has been deprecated in favor of EmitTypedEvent()

https://pkg.go.dev/github.com/cosmos/cosmos-sdk/[email protected]#EventManager.EmitTypedEvent

• Create event types for all our existing events, look at cosmos-sdk modules for examples, they should be a proto message definition
• Refactor existing events everywhere EmitEvent() is called in the gravity module
• Consider refactoring the event type strings, e.g. we have mixed usage of MsgSendToCosmos and BridgeDeposit, making consistent types will go a long way for reducing questions from frontend integrators

The Orchestrator stores private keys in the keys.json file with 644 permissions. That file is stored in the
.gbt directory with 755 permissions inside the user directory. These permissions give any user on the
host access to read the private keys.

The challenge here is maintaining cross-compatbility as the default files api does not expose file permissions for obvious cross-platform reasons.

This issue is the first step in adding NFT support to Gravity Bridge. This part can happen independently of large Cosmos ecosystem NFT support.

All function names are only suggestions of course.

• Create GravityNFT.sol in /solidity/contracts
• GravityNFT constructor should accept the Gravity.sol address and save it as a local parameter
• A function sendNFTToCosmos should validate the NFT, then emit an event with all the details of that NFT
• A function withdrawNFT should accept signatures from the Gravity Bridge, verify them and then send the NFT to the provided destination, this can be copied from Gravity.sol for the most part
• In the unit tests using OpenZepplin to publish a few ERC721 contracts and test every throw condition of GravityNFT.sol, in this case the signature tests can probably be lifted wholesale from existing unit tests.

After this point we can review and merge the basic contract and move our focus to the rust code already open in pr #45 it's only then that we get to the most challenging part of this entire endeavor which is figuring out how to handle the NFT on the cosmos side.

https://github.com/Gravity-Bridge/Gravity-Bridge/runs/5208840100?check_suite_focus=true

In this CI run one of the Orchestrators become confused and got stuck trying to relay an event nonce that was out of order.

Now this probably only happens because of some insane race condition that essentially never happens IRL, the CI test runners operate in a very CPU constrained environments and we see all sorts of extremely strange race conditions there. But just because it's difficult to reproduce does not mean it's not valuable to fix.

Starting this issue to begin discussion on development of NFT transfers and collaboration with Stargaze team.

I (@humanalgorithm) should be considered a dedicated resource to this project as of 1/20/2022.

Note: the description reads from Eth to gravity even though the end goal includes IBC from gravity to stargaze/other ibc chains. We may decide to reach the goal in one or several working pieces.

Relevant parties at the moment:
@jkilpatr @ChristianBorst @shanev @AlfDidNothingWrong

Running bash tests/all-up-test.sh on a Mac with Apple M1 Max processor. Getting error message that eth image can not be found and then failing setup.

qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

See discussion in Discord general channel with @thosmos for further info.

If tx batches are not profitable and volume isn't sufficient going from the cosmos chain to Ethereum, you could end up in a situation where users send SendToEth txs with low fees that keep getting put into batches, but are not profitable so they're never relayed. Making the user wait to cancel the tx in a poor UX and unlikely to actually get timed correctly.

Instead, it would be ideal if a user could bump or replace the fee of an existing tx they already have in a batch.

I've heard rumors this can cause the validator to be instantly removed from the set.

Still a bunch of issues here. I believe some have been addressed, unsure which are still open.

most other SDK chains contain the ability to save the CLI configuration so you don't need to include flags with every command.

Can be done like here:
https://github.dev/cosmos/gaia/blob/cc14c0b5009ecd6da9c869e228467acf729ea81f/cmd/gaiad/cmd/root.go#L55-L56

Gravity Bridge supports evidence based slashing and we even form and send a tx in our integration tests, but there's no way for an end user currently in possession of evidence to submit it.

Support should be added to the Gravity CLI.

Adding this command will losely following the pattern from the other tx's here and mostly be a copy/paste job https://github.com/Gravity-Bridge/Gravity-Bridge/blob/main/module/x/gravity/client/cli/tx.go

We should test upgrading from the previous stable version to the current version in CI. In order to do this we will need to make a new integration test that submits an upgrade proposal, executes that upgrade proposal, swaps out the chain binary and starts the chain again.

Before and after the upgrade a subset of the integration tests should be run. This will be a rather long test.

• Modify the test runner rust code to launch the gravity nodes, Right now this bash loop starts all the orchestrators. It will need to be replaced with a test runner command. This means we'll have three differnet instances of the test runner running at different times. One new one that launches and manages the gravity nodes, one that deploys the contracts and the one actually coordinating the tests. If we where to try and combine any of these we lose the start chains / run tests divide.
• If it detects any other test type the test runner will run the recently built binary. For the case of the upgrad test we can download the latest stable as part of the setup bash script.
• When the test runner code coordinating the upgrade finds that the chain has halted (no new blocks created/block height at upgrade proposal height) it should insert the new binary and start again. If it's any other test it should print an error.
• Finally we run a few more tests on the upgraded chain before calling it a success.

A few things to debate here, should we use the github api to define the latest release? It makes it more automated (test requires no updates in code) but also makes it harder to test upgrades from a specific version. We have some code for parsing the github releases api output in operator tools.

I'd also really like to avoid the three binary split, the test runner instance running the chains not being able to know about the proposal without constantly querying just seems inefficient.

I've noticed that when my ethereum node drops off for a moment the orchestrator will lose connections, which is expected. However when the ethereum node comes back online orchestrator fails to reconnect without a restart.

ERROR gravity_utils::get_with_retry] Failed to get latest block! Is your Eth node working?

Batch timeouts are currently not integration tested. While they are unit tested it's best to get as much coverage as we can

• Add a new test type to the integration test if else tree
• Create a new Rust module for the test
• Use the governance parameter change tools to change the batch timeout parameter to nearly instant
• Manually generate a batch by locking some tokens and requesting the batch utility functions for doing that
• Submit a deposit to complete the timeout process (requires oracle event confirming block height
• Check that all transactions are back in the pool.

Gravity panics when calling send-to-eth with bridgeFee 0. This can also be reproduced if amount is 0.
I did some analyzing and it seems that sdk.ParseTokensNormalized can return empty array of coins which was not checked on gravity side

log when calling:
gravity tx gravity send-to-eth {ETH_ADDRESS} 10token 0token --from validator1 --keyring-backend test --chain-id gravity

panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/Gravity-Bridge/Gravity-Bridge/module/x/gravity/client/cli.CmdSendToEth.func1(0xc001313180, {0xc0011e4e10, 0x3, 0x9})
        /home/sasam/go/src/Ethernal/gravity-bridge/Gravity-Bridge/module/x/gravity/client/cli/tx.go:298 +0x8e5
github.com/spf13/cobra.(*Command).execute(0xc001313180, {0xc0011e4d80, 0x9, 0x9})
        /home/sasam/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0xaf0
github.com/spf13/cobra.(*Command).ExecuteC(0xc00120ea00)
        /home/sasam/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x8b3
github.com/spf13/cobra.(*Command).Execute(0xc00120ea00)
        /home/sasam/go/pkg/mod/github.com/spf13/[email protected]/command.go:902 +0x2f
github.com/spf13/cobra.(*Command).ExecuteContext(0xc00120ea00, {0x2bbf3f8, 0xc001328e40})
        /home/sasam/go/pkg/mod/github.com/spf13/[email protected]/command.go:895 +0x73
github.com/Gravity-Bridge/Gravity-Bridge/module/cmd/gravity/cmd.Execute(0xc00120ea00)
        /home/sasam/go/src/Ethernal/gravity-bridge/Gravity-Bridge/module/cmd/gravity/cmd/root.go:113 +0x1ae
main.main()
        /home/sasam/go/src/Ethernal/gravity-bridge/Gravity-Bridge/module/cmd/gravity/main.go:13 +0x3a

We have quite a few events which don't make the most sense. Gadost recently made an awesome conversion PR to change our deprecated EmitEvent() code to EmitTypedEvent(), but I feel we still need to refactor the event contents.

CancelSendToEth()

• RemoveFromOutgoingPoolAndRefund creates its own event indicating the actual outgoing tx was cancelled, but this one creates an event noting that the message was executed
• Why is there an attribute named "module" whose value is always the message's type? I would suggest changing these attributes to have the key "message", since they are documenting which type of message was processed by the msg_server.

ConfirmLogicCall()

• Gadost updated this to be a EventModule type of event because of the "module" attribute, shouldn't it be something else? confirmHandlerCommon will create a claim event for us, but the EventModule type doesn't make any sense to me.
• Gadost's new EventBatchNonce (the event records a new batch nonce has been created) might be better named EventBatchCreated
• CancelOutgoingTxBatch has OutgoingBatchID with just the nonce value, would it be better to put contract address + nonce for this value instead? If this change is adopted then BuildOutgoingTxBatch should have the same change made.

TODO: Generally look through all the code emitting events + the keys/attributes associated with the events, and improve clarity

The following does not belong in a keeper package, rather in an app package and causes the inability to use the gravity module as a dep.

https://github.com/Gravity-Bridge/Gravity-Bridge/blob/main/module/x/gravity/keeper/addr_prefixes.go

I would like to have some options in what regards to querying attestations (/gravity.v1.Query/GetAttestations):

• ASC/DESC ordering by nonce (currently is ASC)
• observed true/false
• get by specific event nonce
• get by claim type
• get by height

This is especially helpful when tracking the state of a transaction (and also to watch the state of the network's relayers)

All load / store functions as well as all iterators should have unit tests. See this commit message ecc033d

When sending a MsgCancelSendToEth tx, the tx passes validation but fails the call to SendCoinsFromModuleToAccount.

This is because it assumes the token to be refunded is a native Ethereum ERC20: https://github.com/Gravity-Bridge/Gravity-Bridge/blob/main/module/x/gravity/keeper/pool.go#L132. When you attempt to cancel a native Cosmos token, it will fail with something like:

transfer vouchers: 0gravity0xc0a4Df35568F116C370E6a6A6022Ceb908eedDaC is smaller than 125633118gravity0xc0a4Df35568F116C370E6a6A6022Ceb908eedDaC: insufficient funds (at msg 0)

What it should do is something like: https://github.com/Gravity-Bridge/Gravity-Bridge/blob/main/module/x/gravity/keeper/attestation_handler.go#L105-L107

This occurred on a figment node, but only one and not a validator. My first instinct upon hearing this is that they called an RPC endpoint which causes a crash, but the stack trace does not support this, instead it shows the usual validator set generation as part of the endblocker, but if that's the case every node should have crashed since they are in consensus and running the same software.

This behavior looks exactly as if the node was not upgraded. so maybe the logs are wrong about the version?

Feb 02 12:22:58 gravity-1.4.0[29471]: 12:22PM INF validator jailed module=x/staking validator=gravityvalcons145usrtfjvnp8rfy9mnfkqjevr7pex3654864y3
Feb 02 12:22:58 gravity-1.4.0[29471]: 12:22PM INF slashing at current height; not scanning unbonding delegations & redelegations height=683100 module=x/staking
Feb 02 12:22:58 gravity-1.4.0[29471]: 12:22PM INF burned tokens from module account amount=1000ugraviton from=bonded_tokens_pool module=x/bank
Feb 02 12:22:58 gravity-1.4.0[29471]: 12:22PM INF validator slashed by slash factor burned=1000 module=x/staking slash_factor=0.001000000000000000 validator=gravityvaloper17qqfyynlqakqu4gv6mqvvvat98zaqqnrz6pq7r
Feb 02 12:22:58 gravity-1.4.0[29471]: 12:22PM INF validator jailed module=x/staking validator=gravityvalcons1sw6xaywcu066h79pkgcnaw7dse4l4t26a3mh4m
Feb 02 12:22:58 gravity-1.4.0[29471]: panic: generated invalid valset: invalid members: empty
Feb 02 12:22:58 gravity-1.4.0[29471]: goroutine 1 [running]:
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/Gravity-Bridge/Gravity-Bridge/module/x/gravity/keeper.Keeper.GetCurrentValset({{0x5589d2bd7bb0, 0xc001dd1210}, {{0x5589d2c2aa48, 0xc001d8f250}, 0xc000558548, {0x5589d2bd7bb0, 0xc001dd11a0}, {0x5589d2bd7c00, 0xc001dd1230}, {0xc00013ade0, ...}, ...}, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /tmp/tmp.wdP5jdicdJ/buildsources/x/gravity/keeper/keeper_valset.go:307 +0x78a
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/Gravity-Bridge/Gravity-Bridge/module/x/gravity.createValsets({{0x5589d2c08de8, 0xc000052198}, {0x5589d2c2ea80, 0xc00188ea40}, {{0xb, 0x0}, {0xc003440fe0, 0x10}, 0xa6c5c, {0x286a5939, ...}, ...}, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /tmp/tmp.wdP5jdicdJ/buildsources/x/gravity/abci.go:38 +0x178
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/Gravity-Bridge/Gravity-Bridge/module/x/gravity.EndBlocker({{0x5589d2c08de8, 0xc000052198}, {0x5589d2c2ea80, 0xc00188ea40}, {{0xb, 0x0}, {0xc003440fe0, 0x10}, 0xa6c5c, {0x286a5939, ...}, ...}, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /tmp/tmp.wdP5jdicdJ/buildsources/x/gravity/abci.go:18 +0x2bb
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/Gravity-Bridge/Gravity-Bridge/module/x/gravity.AppModule.EndBlock({{}, {{0x5589d2bd7bb0, 0xc001dd1210}, {{0x5589d2c2aa48, 0xc001d8f250}, 0xc000558548, {0x5589d2bd7bb0, 0xc001dd11a0}, {0x5589d2bd7c00, 0xc001dd1230}, ...}, ...}, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /tmp/tmp.wdP5jdicdJ/buildsources/x/gravity/module.go:171 +0x9b
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/cosmos/cosmos-sdk/types/module.(*Manager).EndBlock(_, {{0x5589d2c08de8, 0xc000052198}, {0x5589d2c2ea80, 0xc00188ea40}, {{0xb, 0x0}, {0xc003440fe0, 0x10}, 0xa6c5c, ...}, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:481 +0x375
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/Gravity-Bridge/Gravity-Bridge/module/app.(*Gravity).EndBlocker(_, {{0x5589d2c08de8, 0xc000052198}, {0x5589d2c2ea80, 0xc00188ea40}, {{0xb, 0x0}, {0xc003440fe0, 0x10}, 0xa6c5c, ...}, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /tmp/tmp.wdP5jdicdJ/buildsources/app/app.go:651 +0xbb
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc000102d00, {0xa6c5c})
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:210 +0x37f
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc00b6a84e0, {0xa6c5c})
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/tendermint/[email protected]/abci/client/local_client.go:288 +0x10f
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc0059c5720, {0xa6c5c})
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/tendermint/[email protected]/proxy/app_conn.go:89 +0x5e
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x5589d2c09b78, 0xc00b6a9200}, {0x5589d2c1c8d0, 0xc0059c5720}, 0xc00000c5a0, {0x5589d2c292e8, 0xc0059c5670}, 0x77074)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:327 +0x848
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc00b7ab426, 0x7}}, {0xc00b7ab470, 0x10}, 0x77074, 0xa6c5b, {{0xc0052e3300, ...}, ...}, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:140 +0x2ac
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc00b7ab426, 0x7}}, {0xc00b7ab470, 0x10}, 0x77074, 0xa6c5b, {{0xc0052e3300, ...}, ...}, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:503 +0x2b3
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc00b7ab426, 0x7}}, {0xc00b7ab470, 0x10}, 0x77074, 0xa6c5b, {{0xc0052e3300, ...}, ...}, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:416 +0xcdb
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc0017f01a0, {0x5589d2c2fd80, 0xc000c080d0})
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:268 +0x705
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc00b7ab426, 0x7}}, {0xc00b7ab470, 0x10}, 0x77074, 0xa6c5b, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/tendermint/[email protected]/node/node.go:325 +0x135
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/tendermint/tendermint/node.NewNode(0xc000217e00, {0x5589d2be7ea0, 0xc000c0f540}, 0xc000568960, {0x5589d2bc9f60, 0xc004882678}, 0xc0005689d0, 0x5589d2ba1fd0, 0xc000568ac0, {0x5589d2c09b78, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/tendermint/[email protected]/node/node.go:733 +0xcdc
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x5589d2c11b88, 0xc0010bb3c0}, ...}, ...)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:263 +0xa58
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc0001fef00, {0xc000398390, 0x0, 0x1})
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/cosmos/[email protected]/server/start.go:129 +0x238
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/spf13/cobra.(*Command).execute(0xc0001fef00, {0xc000398380, 0x1, 0x1})
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0xaf0
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000d14280)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x8b3
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/spf13/cobra.(*Command).Execute(0xc000d14280)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/spf13/[email protected]/command.go:902 +0x2f
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/spf13/cobra.(*Command).ExecuteContext(0xc000d14280, {0x5589d2c08e58, 0xc00013c7b0})
Feb 02 12:22:58 gravity-1.4.0[29471]:         /go/pkg/mod/github.com/spf13/[email protected]/command.go:895 +0x73
Feb 02 12:22:58 gravity-1.4.0[29471]: github.com/Gravity-Bridge/Gravity-Bridge/module/cmd/gravity/cmd.Execute(0xc000d14280)
Feb 02 12:22:58 gravity-1.4.0[29471]:         /tmp/tmp.wdP5jdicdJ/buildsources/cmd/gravity/cmd/root.go:113 +0x1ae
Feb 02 12:22:58 gravity-1.4.0[29471]: main.main()
Feb 02 12:22:58 gravity-1.4.0[29471]:         /tmp/tmp.wdP5jdicdJ/buildsources/cmd/gravity/main.go:13 +0x3a
Feb 02 12:22:58 systemd[1]: gravity.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 02 12:22:58 systemd[1]: gravity.service: Failed with result 'exit-code'.

the protobuf generation code should be run in CI and compared with what is currently comitted to the repo.

The 'make proto-gen' makefile script generates the gravity bridge proto files for the Go code, likewise 'cargo run' in orchestrator/proto-gen builds the proto files for the rust code.

Both of these should be run in a github actions job that confirms the git diff between the generated files and the current commit is empty. This will allow us to verify if the proto files have not been updated or modified either maliciously or because the user has an outdated version of buf.

When building a batch, transactions are first collected from the outgoing transaction pool in order to calculate the total bridge fee and check if a more profitable batch of the same token type already exists in the pool. When doing this, there is no check if the translation’s destination address is included in the blacklist or not. Later, when transactions are collected to be included in the batch, additional blacklist filtering is applied. This can cause potential creation of the less profitable batch than the one of the same token type with the biggest nonce already in the pool. Also, the double retrieval of the calculations seems unnecessary.

When gravity-3 started up, our validator was showing 0 voting power before block production started. Normally I would expect to see the validator's correct voting power. In this case once block production started, our voting power displayed correctly.

We are using TMKMS remote signing.

Pre-block production:

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "node_info": {
      "protocol_version": {
        "p2p": "8",
        "block": "11",
        "app": "0"
      },
      "id": "d20fb90c25dcd447fc574d20c3511a05b19aa9a5",
      "listen_addr": "tcp://0.0.0.0:26656",
      "network": "gravity-bridge-3",
      "version": "0.34.14",
      "channels": "40202122233038606100",
      "moniker": "xxTBMytfqF",
      "other": {
        "tx_index": "on",
        "rpc_address": "tcp://0.0.0.0:26657"
      }
    },
    "sync_info": {
      "latest_block_hash": "",
      "latest_app_hash": "",
      "latest_block_height": "0",
      "latest_block_time": "1970-01-01T00:00:00Z",
      "earliest_block_hash": "",
      "earliest_app_hash": "",
      "earliest_block_height": "0",
      "earliest_block_time": "1970-01-01T00:00:00Z",
      "catching_up": false
    },
    "validator_info": {
      "address": "C3A2C9D34E4F603FA7879B4EF4E32A64489DAC22",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "i3OGo7AI+ZU3i48AVk09itEtzQh7njmTCBsUKOedgUs="
      },
      "voting_power": "0"
    }
  }
}

Post Block production:

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "node_info": {
      "protocol_version": {
        "p2p": "8",
        "block": "11",
        "app": "0"
      },
      "id": "d20fb90c25dcd447fc574d20c3511a05b19aa9a5",
      "listen_addr": "tcp://0.0.0.0:26656",
      "network": "gravity-bridge-3",
      "version": "0.34.14",
      "channels": "40202122233038606100",
      "moniker": "xxTBMytfqF",
      "other": {
        "tx_index": "on",
        "rpc_address": "tcp://0.0.0.0:26657"
      }
    },
    "sync_info": {
      "latest_block_hash": "6F942BE68BBDBBD985A55E21C9516D830D4261CC2D1E083EF6B4F99C8DEB2040",
      "latest_app_hash": "C5AB51E5F4090AD323971E04D26E07317D68DB3098E45C31A0DD91D572803B1F",
      "latest_block_height": "487548",
      "latest_block_time": "2022-01-19T02:57:32.276145844Z",
      "earliest_block_hash": "74B590C83F260AEE720A4E303490A1D5E8CC28DCED149B44482A4AADB790B641",
      "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855",
      "earliest_block_height": "487540",
      "earliest_block_time": "2021-12-10T14:45:49.658958945Z",
      "catching_up": false
    },
    "validator_info": {
      "address": "C3A2C9D34E4F603FA7879B4EF4E32A64489DAC22",
      "pub_key": {
        "type": "tendermint/PubKeyEd25519",
        "value": "i3OGo7AI+ZU3i48AVk09itEtzQh7njmTCBsUKOedgUs="
      },
      "voting_power": "36445895"
    }
  }

This will allow us to return more conclusive error messages by checking the delegate keys for that particular provided address.

Porting discussion over from "Fix Mac M1 Build"

There is a "Docker not found" error when running ./start-chains.sh

Update: I am running Gravity on a Fedora box and I can confirm seeing the same error on Docker build:

/usr/bin/which: no docker in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/go/bin:/usr/local/go/bin)

relay_market test fails due to empty ALCHEMY_ID

[3](https://github.com/Gravity-Bridge/Gravity-Bridge/runs/5200273450?check_suite_focus=true#step:4:3)
  shell: /usr/bin/bash -e {0}
[4](https://github.com/Gravity-Bridge/Gravity-Bridge/runs/5200273450?check_suite_focus=true#step:4:4)
  env:
[5](https://github.com/Gravity-Bridge/Gravity-Bridge/runs/5200273450?check_suite_focus=true#step:4:5)
    ALCHEMY_ID: 
[6](https://github.com/Gravity-Bridge/Gravity-Bridge/runs/5200273450?check_suite_focus=true#step:4:6)
+++ dirname tests/all-up-test.sh

runner should printing
ALCHEMY_ID: *** if value present

9:01AM INF asserting crisis invariants inv=8/12 module=x/crisis name=gravity/module-balance
panic: invariant broken: Mismatched balance of ibc/D956638E52570976F762354757863F60F2AFE4879E849CB1C77B163F0BF11C64 actual balance 200002 expected balance 0
	CRITICAL please submit the following transaction:
		 tx crisis invariant-broken gravity module-balance

goroutine 70 [running]:
github.com/cosmos/cosmos-sdk/x/crisis/keeper.Keeper.AssertInvariants(0xc000e32c80, 0xc, 0x10, 0x2d8ced8, 0xc001084590, 0xc00027e470, 0x2d5daf8, 0xc0010a64f0, 0x2d5db48, 0xc0010a6570, ...)
	/home/justin/go/pkg/mod/github.com/cosmos/[email protected]/x/crisis/keeper/keeper.go:83 +0xb2f
github.com/cosmos/cosmos-sdk/x/crisis.EndBlocker(0x2d79688, 0xc00019c168, 0x2d8f120, 0xc0047d6240, 0xb, 0x0, 0xc001345400, 0x10, 0xa781d, 0x152a5b9b, ...)
	/home/justin/go/pkg/mod/github.com/cosmos/[email protected]/x/crisis/abci.go:20 +0x278
github.com/cosmos/cosmos-sdk/x/crisis.AppModule.EndBlock(0xc001334e60, 0x203000, 0x2d79688, 0xc00019c168, 0x2d8f120, 0xc0047d6240, 0xb, 0x0, 0xc001345400, 0x10, ...)
	/home/justin/go/pkg/mod/github.com/cosmos/[email protected]/x/crisis/module.go:168 +0xf8
github.com/cosmos/cosmos-sdk/types/module.(*Manager).EndBlock(0xc00120f9d0, 0x2d79688, 0xc00019c168, 0x2d8f120, 0xc0047d6240, 0xb, 0x0, 0xc001345400, 0x10, 0xa781d, ...)
	/home/justin/go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:481 +0x23d
github.com/Gravity-Bridge/Gravity-Bridge/module/app.(*Gravity).EndBlocker(0xc001243d10, 0x2d79688, 0xc00019c168, 0x2d8f120, 0xc0047d6240, 0xb, 0x0, 0xc001345400, 0x10, 0xa781d, ...)
	/home/justin/go/src/github.com/cosmos/gravity-bridge/module/app/app.go:659 +0xc5
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc0001ae1a0, 0xa781d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/justin/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:210 +0x37d
github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc000597c20, 0xa781d, 0x0, 0x0, 0x0)
	/home/justin/go/pkg/mod/github.com/tendermint/[email protected]/abci/client/local_client.go:288 +0x106
github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc00125c300, 0xa781d, 0x0, 0x0, 0x0)
	/home/justin/go/pkg/mod/github.com/tendermint/[email protected]/proxy/app_conn.go:89 +0x6d
github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x2d7a418, 0xc0012d9b00, 0x2d854b8, 0xc00125c300, 0xc000fbf4a0, 0x2d8ca88, 0xc0007a7720, 0x77074, 0x0, 0x0, ...)
	/home/justin/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:327 +0xc02
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc0019e8620, 0xb, 0x0, 0xc005a51db0, 0x7, 0xc005a51dc0, 0x10, 0x77074, 0xa781c, 0xc00005d2e0, ...)
	/home/justin/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:140 +0x2ea
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc000f54fc0, 0xc00026bb00)
	/home/justin/go/pkg/mod/github.com/tendermint/[email protected]/blockchain/v0/reactor.go:398 +0xf25
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
	/home/justin/go/pkg/mod/github.com/tendermint/[email protected]/blockchain/v0/reactor.go:110 +0xb5

Like before this probably has to do with the way we handle tx's moving between the pool and batches than an actual failure. But we should investigate and correct the issue.