It seems that the consensus key generated on the YubiHSM is different in size than the one generated by a simple gaiad command:
gaiad version: cosmoshub-test-stargate
tmkms version: 0.10.0-alpha7

Compare size:

cosmosvalconspub1zcjduepq4f2zzkjr20v4gt0gy9nmdznxsvmgluyn3vnjmgqq58q9t47uyffqndgy0p
cosmosvalconspub1wzmcs4kpsetyk7fn7czvdljgawa93r6mnzvcwt9a366rep2gcs0q62ce7v

gaiad tendermint show-validator
cosmosvalconspub1zcjduepq4f2zzkjr20v4gt0gy9nmdznxsvmgluyn3vnjmgqq58q9t47uyffqndgy0p

tmkms yubihsm keys list
- 0x0001: [cons] cosmosvalconspub1wzmcs4kpsetyk7fn7czvdljgawa93r6mnzvcwt9a366rep2gcs0q62ce7v
   label: "cosmosvalconspub:2019-03-12T21:03:43Z"
- 0x0002: [cons] FE4AB0236C7B8E3C3687D7703CF58FD16128ABF28A17CF89F585C7D724D1B0E7
   label: "cosmoshub-test-stargate-b"

The first key was generated in the amino times, but the second key showed a same-size consensus key at generation.

Additionally, you can see that if a random label is used, then the key is not printed with the correct bech32 encoding.

Currently the protocol used by a "Transaction Microservice" as defined in [[tx_signer.source]] in tmkms.toml is a bespoke JSON-over-HTTP RPC protocol.

This issue tracks adding support for using gRPC as an alternative/replacement. Ideally this would leverage using Protocol Buffers schema definitions defined upstream in Cosmos SDK:

cosmos/cosmos-sdk#6213

Hey,

Wanted to bring back an issue.

I've done some work on a tendermint signer using SGX for tendermint 0.33.x here. It was mainly as a PoC, but now that it's working (and was pretty simple to set up), I thought it might be interesting to polish it up and port it into a more mature KMS framework, like tmkms.

Would you guys be interested in adding SGX as a signing provider? I'm not a rust expert, but I think I could get the port done without too much trouble (though I would appreciate pointers on which interfaces would need to be implemented).

Anyways, I'd be happy to brainstorm on this with you guys if you're open to it.

The implementation of the function TxSigner::sign_bytes simply returns the internal string representation of a SignMsg object. The documentation of the struct clearly states “String representation of a message that describes a particular transaction to be signed by transaction signer.
It is unclear to a user whether or not if the returned data is signed or if it is to-be signed. This misconception may lead to data being treated signed when it is not, or vice-versa.
Recommendation
Return signed data or explain properly that the data returned is not actually signed.

I have an issue syncing a Kava validator.
Not sure if this is tmkms issue or tendermint issue but the problem only manifests when tmkms is being used.

When I configure the node to use tmkms is will fast sync until the latest block but then immediately start lagging behind the chain.
I can't see any errors in the node logs.

If I run tmkms with the verbose flag I can see a lot of request and response for the public key going on (a few dozens a second)
Not sure if this is what's causing the issue or just normal behavior.

If I configure the node to not use a remote signer at all it syncs up an does not lag behind
Any help debugging and solving it will be appreciated

Presently only plaintext HTTP connections are supported from the KMS to the transaction source / microservice. This is a tracking issue for adding HTTPS support.

The HTTP client is hyper. It should be fairly easy to add HTTPS support using hyper-rustls.

NOTE: this issue was originally opened as tendermint/tmkms#386

Presently Tendermint KMS only supports signing consensus proposals/votes. That said, the prerequisites are all in place, both in the tendermint crate and the yubihsm crate, to sign data using account keys as well.

This proposal is to add first class support to sign transactions in the format used by the Cosmos SDK. Though this project is "Tendermint KMS" and intended to be agnostic to specific Tendermint applications, the Cosmos SDK's transaction format seems widely adopted among Tendermint applications, enough to warrant this support. (In the future there are various ways this support could be generalized to signing other transaction/object types, using e.g. WASM, but that sort of approach is considered out-of-scope for this particular proposal)

Motivation

Our initial use case at iqlusion is signing transactions for a Terra exchange rate oracle. The KMS-side goal of this proposal is to avoid making the KMS-side functionality hardcoded to this particular application, but rather to place a set of constraints (provided via tmkms.toml) on the transactions to be signed, and having the KMS assert those constraints are being met. Ideally this will allow the KMS to provide hardware-backed signing functionality for any type of Cosmos transaction, outsourcing the transaction composition to a separate service, but still allowing the KMS to constrain aspects of the signed transaction.

High-level Design

There are several ways such a feature could be implemented. The proposal below is one, with debatable details, which presently targets doing things as similarly to the existing Tendermint consensus signing as possible:

• KMS is configured with a key usage policy and the address of a transaction-building service to connect to. Connection is encrypted using Secret Connection.
• Transaction-building service sends a newly defined Amino-encoded request (e.g. SignTxRequest) to the KMS containing a transaction (or declaration of the desired transaction, with actual serialization handled by the KMS itself)
• KMS validates the requested transaction according to a configured policy, and if the transaction is authorized according to that policy, signs the transaction (using either a software-backed or YubiHSM2-backed key. It may be possible to also support Ledger, but that might involved changes to the Ledger validator application)

Design Rationale

Here's a brief rationale and some alternatives to consider for each of these options:

• Outbound Secret Connection: this follows the current approach used for connecting to validators. I still think it would be interesting for the KMS to support both inbound and outbound connections (as is otherwise presently possible with P2P traffic) which users can configure to their liking. Additionally there are many other options for transport encryption to consider (TLS, Noise, libp2p+Noise). However, using an outbound Secret Connection allows reuse of all of the existing networking code, and in doing so, means that we can also separately tackle the issues of different connection models or different transport encryption in a single place (in addition to moving to e.g. Tokio), rather than trying to shoehorn that work onto this proposal.
• Amino: ...is not long for this world. Protobufs are probably the main alternative to consider. However, right now both tendermint-rs as well as tmkms itself import the main Protobuf library we'd like to use, Prost, in a way which precludes using it for proper Protobufs. We could pull in an additional Protobuf library now, but the other alternative there is to just use Amino for now and switch things over to Protobufs when Tendermint itself makes that transition. Once this has been addressed it would also be possible to consider e.g. gRPC libraries based on Prost, like Tonic.

Open Questions

1. What restrictions should the KMS support on transaction signing?
2. Could Ledgers be supported in addition to YubiHSM and soft sign backends?
3. Is there a way to support generic signing with restrictions (i.e. not a pure signing oracle) without coupling directly to the Cosmos transaction format?

tempdir crate has been deprecated; use tempfile instead

The tempdir crate has been deprecated
and the functionality is merged into tempfile.

See advisory page for additional details.

Hi all,

I'm trying to use tmkms to generate secret_connection.key following this guide:
https://hub.cosmos.network/master/validators/kms/kms_ledger.html

But I got this error message:

lecong@latitude:~/workspace/COSMOS/tmkms$ ./target/release/tmkms keygen secret_connection.key
error: unrecognized command `keygen`

tmkms 0.7.2
Tony Arcieri <[email protected]>, Ismail Khoffi <[email protected]>
Tendermint Key Management System

SUBCOMMANDS:
    help       show help for a command
    start      start the KMS application
    version    display version information
    ledger     subcommands for Ledger

Please help to give me some advice, thank you so much

Hi, after following the steps in README.yubihsm.md I still get the following error when running tmkms yubihsm detect -v (on ubuntu 20.04):
2021-04-20T14:45:34.129126Z DEBUG abscissa_core::component::registry: registered component: abscissa_core::terminal::component::Terminal (v0.6.0-pre.1) 2021-04-20T14:45:34.129155Z DEBUG abscissa_core::component::registry: registered component: abscissa_core::trace::component::Tracing (v0.6.0-pre.1) 2021-04-20T14:45:34.130652Z DEBUG yubihsm::connector::usb::device: USB: enumerating devices... 2021-04-20T14:45:34.130658Z DEBUG yubihsm::connector::usb::device: USB(bus=1,addr=2): found YubiHSM device error: couldn't detect USB devices: USB error: USB(bus=1,addr=2): error opening device: Access denied (insufficient permissions)

I previously added a new group with sudo addgroup yubihsm from "user1" and added user1 to the group "yubihsm" with sudo adduser user1 yubihsm

I ran udevadm control --reload-rules && udevadm trigger and also rebooted.

What could be the issue? Thanks

Following previous conversations it seems that this is the best way to try to deploy an "HA" configuration:

• one TMKMS (Active) connected to multiple validatords in the same chain-id
• keep a second TMKSM (Passive)

tendermint/tmkms#272

Is it safe?
Have any further developments or improvement been made?

when updating from 0.7.2 to 0.7.3 via cargo install-update -a

fixed by rustup update to rust version 1.44.1 (c7087fe00 2020-06-17)

Would it be possible to have a prometheus endpoint built into tmkms for monitoring. This would allow us to better monitor the KMS module, as at the moment we're monitoring it through its logs.

Some ideas that could be useful:

Counters for:
signed_prevotes
signed_precommits
signed_proposals
timeouts/errors

Other:
Number of validator_connections
hsm_delay time/histogram
chain block heights

Would appreciate your thoughts and possibility of this, thank you!

The ledger-validator-app presently uses Amino encoding. Furthermore, various support code in TMKMS does as well, e.g.:

https://github.com/iqlusioninc/tmkms/blob/develop/src/commands/ledger.rs

The app itself needs to be modified to support the new Protobuf canonicalization/signature format, and then support needs to be added to the TMKMS host-side driver and commands.

Is there any way to check if tmkms is starting and working properly?
Is the only way to check the log at present?

Thank you in advance!

tmkms can't access desmos v0.3.0 for validator.
also it can't access regen-network wasmd v0.7.1

Perhaps tmkms doesn't seem to be able to access the project that used for cosmos-sdk v0.38.1.

The URL below has detailed reporting.
desmos-labs/desmos#107

I'm not sure if this is a Tendermint issue or tmkms issue but if I try to associate the validator node id
with tmkms the id keep changing randomly everytime I restart the validator.

This is testend on the latest Kava node (0.11.0-alpha.1) with Tendermint v0.33.6

If I run tmkms without the peer id.
I get the log -

Sep 17 07:20:54.056 WARN tmkms::session: [kava-testnet-9000@tcp://172.17.0.2:26658]: unverified validator peer ID! 
(D61B22B4673E5174B663DA6F7398702985A6AFA7)

I than set the peer id to my toml file and restart tmkms -
addr = "tcp://[email protected]:26658"

This will work as long as the validator is running but if I restart the validator tmkms will start showing the following error and refuse to connect.

Sep 17 07:05:17.139 ERROR tmkms::client: [kava-testnet-9000@tcp://172.17.0.2:26658] verification failed: 172.17.0.2:26658: 
validator peer ID mismatch! (expected D61B22B4673E5174B663DA6F7398702985A6AFA7, 
got FD4AB758EFA25547110D651D44E748133A55718A)

Also I couldn't find how to do the opposite (get the validator verifying tmkms ID)

When I try to set validator config,json with priv_validator_laddr as follows
priv_validator_laddr = "tcp://[email protected]:26658"
where 5E56C6946AD644795E3EE948FA224C1A10A51A61 is the node ID I'm getting from the log for KMS node ID

I get the following error when starting the validator -

ERROR: error with private validator socket client: failed to start private validator: listen tcp: 
lookup [email protected]: no such host

I guess I'm just not setting the ID at the right place but I couldn't find where I should set it

This is a tracking issue for improvements to the tx-signer feature which are possible following the Stargate upgrade.

• Protobuf Transaction Support in stdtx crate: iqlusioninc/crates#459
• gRPC Transaction Sources: #97
• Replace [[tx_signer.seq_file]] with on-chain query

This roadmap is still TBD, so please subscribe for updates!

HEADS UP: THE LEDGER VALIDATOR APP WILL NOT WORK ON TENDERMINT V0.34 NETWORKS INCLUDING STARGATE

Note that the Ledger Validator App performs consensus signing and is different from the Ledger Cosmos App which is a wallet application unrelated to TMKMS.

After some discussion about what to do about #212, we have come to the conclusion that there is little interest in the Ledger Validator App, nor have we been able to find anyone who is actually using it in production.

To update the app for Tendermint v0.34/Stargate compatibility (which uses new message formats and signing algorithms), we would need to find financing to hire a Ledger developer to do the update work. This has not happened and there seems to be a general lack of interest.

Based on that, barring any input to the contrary and financing of the app's ongoing development to retain compatibility, we intend to deprecate Ledger support and remove it from a future TMKMS release.

This is a tracking issue for supporting account keys held by a Ledger device.

Since Ledger apps which sign Tendermint transactions today (e.g. the Cosmos App) typically require user interaction, and KMS transaction signing is specifically intended to be used for online, non-interactive signing, the first step towards adding support for this feature would likely involve adding account key support to the Ledger Tendermint Validator App which provides non-interactive account key support.

Upgrading from v0.7.1 to v0.7.2, bumped into

tried uninstalling then reinstall. same issue.

$ cargo -V                                                                                                                                                                                                                   
cargo 1.39.0 (1c6ec66d5 2019-09-30)
$ rustc -V                                                                                                                                                                                                                   
rustc 1.39.0 (4560ea788 2019-11-04)

See here for solution.

secret_key = "path/to/secret_connection.key"

crate has been renamed to block-cipher

This crate has been renamed from block-cipher-trait to block-cipher.

The new repository location is at:

<https://github.com/RustCrypto/traits/tree/master/block-cipher>

See advisory page for additional details.

cargo install tmkms --features=yubihsm

rust-lang/rust#49146

Sending in /tmp/cargo-installA2L1N7 privately to Tony.

rustc 1.44.1 (c7087fe00 2020-06-17)
cargo 1.44.1 (88ba85757 2020-06-11)

I am seeing the following issue with tmkms version 0.6.3 and gaiad version 2.0.8
The signer process will connect to a cosmos validator and logs the following:

[cosmoshub-3@tcp://] signed PreCommit:<nil> at h/r/s 1606522/0/2 (104 ms)
[cosmoshub-3@tcp://] signed PreCommit:<nil> at h/r/s 1606523/0/2 (104 ms)
[cosmoshub-3@tcp://] signed PreCommit:<nil> at h/r/s 1606524/0/2 (105 ms)
``

The validator acknowledges a connection with tmkms, starts syncing, and claims to be pushing votes. However, it never signs any blocks and tmkms always shows nil block id.

I've exported encrypted backup from yubihsm2 to .enc file ( with $ tmkms yubihsm keys export ...) and trying to decrypt it with wrapkey using yubihsm-shell:

yubihsm> set informat base64
yubihsm> decrypt aesccm 1 0x0001 "encrypted key from .enc file here with == at the end"
Failed to decrypt data: Malformed command / invalid data

It's a test key, so it's not a problem to decrypt the backup outside the HSM using its wrap key (I'm trying to convert yubihsm2 key to softsign tmkms key format). Could you please tell what's wrong with decryption? Or is there another method to convert yubihsm2 key to tmkms softsign?

Running on v0.7.2 -- after adding a second key to the HSM, I got:

tmkms::client: ... invalid key: expected only one key in keyring

It is my understanding that the config part auth = { key = 2 ... was to define the key index, is that not the case?

Is this just something that hasn't been implemented yet?

net2 crate has been deprecated; use socket2 instead

The net2 crate has been deprecated
and users are encouraged to considered socket2 instead.

See advisory page for additional details.

Does this hold the keys, and deliver signed transactions upon request, or does it connect to the peer-to-peer network and act as a validator?

I'm interested in using this so that I can have sentries on akash, and validators in my lair where I have openmptcprouter going and theoretically good connectivity to the planet.

I'm also interested in using this so that I can validate networks like Juno from Raspberry Pi devices even though the smart contracts aren't capable of running on ARM. My personal hunch is that a validator should become a tangible object.

Software like this could potentially allow a single Raspberry Pi to validate multiple networks although I'm not sure if that's desirable from a decentralization / reliability perspective.

I'm going to try it out later today but as I'm sure you guys know trying it out is going to take some time and I figured that I would ask before I do that.

So since this is a complex topic and stuff, let me try to boil it down-- could I use this software like:

3 sentries in akash
1 tmkms instance

Thanks a lot for any help and pointers, I've come to the realization recently that if one were only signing, the validator could be positively teeny tiny, we could be talking about single core devices and the less complex the validator actually is then the more secure it can be.

*Haven't checked on that single core claim. Basically collecting hunches here.

Hi,
I'm getting the following error when I try to install tmkms with cargo.
a@a-VirtualBox:~/home/a/src/github.com/tendermint/tendermint/tools/tm-signer-harness$ cargo install tmkms Updating crates.io index Installing tmkms v0.7.2 Downloaded signal-hook v0.1.15 Downloaded syn v1.0.20 Compiling proc-macro2 v1.0.12 Compiling unicode-xid v0.2.0 Compiling syn v1.0.20 Compiling libc v0.2.69 Compiling typenum v1.12.0 Compiling cfg-if v0.1.10 Compiling cc v1.0.41 Compiling byteorder v1.3.4 Compiling subtle v2.2.2 Compiling getrandom v0.1.14 Compiling autocfg v1.0.0 Compiling lazy_static v1.4.0 Compiling fnv v1.0.6 Compiling log v0.4.8 Compiling memchr v2.3.3 Compiling byte-tools v0.3.1 Compiling serde v1.0.110 Compiling fake-simd v0.1.2 Compiling opaque-debug v0.2.3 Compiling rustc-demangle v0.1.16 Compiling bytes v0.5.4 Compiling failure_derive v0.1.8 Compiling futures-core v0.3.5 Compiling rand_core v0.4.2 Compiling itoa v0.4.5 Compiling slab v0.4.2 Compiling proc-macro2 v0.4.30 Compiling unicode-xid v0.1.0 Compiling pin-project-lite v0.1.5 Compiling regex-syntax v0.6.17 Compiling strsim v0.9.3 Compiling ident_case v1.0.1 Compiling maybe-uninit v2.0.0 Compiling futures-task v0.3.5 Compiling ryu v1.0.4 Compiling subtle v1.0.0 Compiling futures-sink v0.3.5 Compiling httparse v1.3.4 Compiling bitflags v1.2.1 Compiling pin-utils v0.1.0 Compiling nix v0.14.1 Compiling either v1.5.3 Compiling arc-swap v0.4.6 Compiling stable_deref_trait v1.1.1 Compiling try-lock v0.2.2 Compiling remove_dir_all v0.5.2 Compiling ansi_term v0.11.0 Compiling tower-service v0.3.0 Compiling semver-parser v0.7.0 Compiling ppv-lite86 v0.2.6 Compiling void v1.0.2 Compiling termcolor v1.1.0 Compiling canonical-path v2.0.2 Compiling uuid v0.8.1 Compiling once_cell v1.3.1 Compiling generational-arena v0.2.7 Compiling backtrace-sys v0.1.37 Compiling clear_on_drop v0.2.3 Compiling secp256k1-sys v0.1.2 Compiling num-traits v0.2.11 Compiling num-integer v0.1.42 Compiling indexmap v1.3.2 Compiling tracing-core v0.1.10 Compiling thread_local v1.0.1 Compiling block-padding v0.1.5 Compiling futures-channel v0.3.5 Compiling rand_core v0.3.1 Compiling http v0.2.1 Compiling itertools v0.7.11 Compiling owning_ref v0.4.1 Compiling http-body v0.3.1 Compiling quote v1.0.4 Compiling iovec v0.1.4 Compiling time v0.1.43 Compiling net2 v0.2.34 Compiling atty v0.2.14 Compiling rand v0.4.6 Compiling signal-hook-registry v1.2.0 Compiling wait-timeout v0.2.0 Compiling generic-array v0.12.3 Compiling generic-array v0.9.0 Compiling regex-automata v0.1.9 Compiling prost-amino v0.5.0 Compiling tracing-log v0.1.1 Compiling want v0.3.0 Compiling aho-corasick v0.7.10 Compiling quote v0.6.13 Compiling smallvec v0.6.13 Compiling rand_core v0.5.1 Compiling mio v0.6.22 Compiling tempdir v0.3.7 Compiling signal-hook v0.1.15 Compiling digest v0.8.1 Compiling block-buffer v0.7.3 Compiling crypto-mac v0.7.0 Compiling universal-hash v0.3.0 Compiling stream-cipher v0.3.2 Compiling aead v0.2.0 Compiling digest v0.7.6 Compiling backtrace v0.3.46 Compiling secp256k1 v0.17.2 Compiling matchers v0.0.1 Compiling regex v1.3.7 Compiling syn v0.14.9 Compiling synstructure v0.12.3 Compiling darling_core v0.10.2 Compiling serde_derive v1.0.110 Compiling pin-project-internal v0.4.16 Compiling gumdrop_derive v0.7.0 Compiling tracing-attributes v0.1.7 Compiling thiserror-impl v1.0.16 Compiling rand_chacha v0.2.2 Compiling tokio v0.2.20 Compiling atomicwrites v0.2.5 Compiling curve25519-dalek v1.2.3 Compiling sha2 v0.8.1 Compiling hmac v0.7.1 Compiling poly1305 v0.5.2 Compiling color-backtrace v0.3.0 Compiling zeroize_derive v1.0.0 Compiling signature_derive v1.0.0-pre.2 Compiling darling_macro v0.10.2 Compiling pin-project v0.4.16 Compiling gumdrop v0.7.0 Compiling tracing v0.1.13 Compiling thiserror v1.0.16 Compiling rand v0.7.3 Compiling tokio-util v0.3.1 Compiling hkdf v0.8.0 Compiling zeroize v1.1.0 Compiling signature v1.0.0-pre.1 Compiling failure v0.1.8 Compiling darling v0.10.2 Compiling chrono v0.4.11 Compiling toml v0.5.6 Compiling semver v0.9.0 Compiling serde_json v1.0.53 Compiling futures-util v0.3.5 Compiling elliptic-curve v0.3.0 Compiling subtle-encoding v0.4.1 Compiling secrecy v0.6.0 Compiling subtle-encoding v0.5.1 Compiling curve25519-dalek v2.0.0 Compiling chacha20 v0.3.4 Compiling hkd32 v0.3.1 Compiling ed25519 v1.0.0-pre.1 Compiling prost-amino-derive v0.5.0 Compiling abscissa_derive v0.5.0 Compiling tracing-subscriber v0.1.6 Compiling tai64 v3.1.0 Compiling h2 v0.2.5 Compiling ed25519-dalek v1.0.0-pre.2 Compiling p384 v0.1.0 Compiling k256 v0.1.1 Compiling p256 v0.1.0 Compiling x25519-dalek v0.6.0 Compiling chacha20poly1305 v0.3.3 Compiling abscissa_core v0.5.2 Compiling hyper v0.13.5 Compiling ecdsa v0.4.0 Compiling signatory v0.18.1 Compiling signatory-secp256k1 v0.18.1 error[E0405]: cannot find trait PrehashSignaturein cratesignature--> /home/a/.cargo/registry/src/github.com-1ecc6299db9ec823/signatory-secp256k1-0.18.1/src/lib.rs:24:10 | 24 | #[derive(Signer)] | ^^^^^^ not found insignature`

error[E0405]: cannot find trait PrehashSignature in crate signature
--> /home/a/.cargo/registry/src/github.com-1ecc6299db9ec823/signatory-secp256k1-0.18.1/src/lib.rs:89:39
|
89 | #[derive(Clone, Debug, Eq, PartialEq, Verifier)]
| ^^^^^^^^ not found in signature

error: aborting due to 2 previous errors

For more information about this error, try rustc --explain E0405.
error: failed to compile tmkms v0.7.2, intermediate artifacts can be found at /tmp/cargo-installIIpaC6

Caused by:
could not compile signatory-secp256k1.
`
Can you tell me how to overcome this error. Thank you very much.

Can somebody provide me a manual for Softsign installation, what should I use as
as validator addr in this case?
I need to setup node as validator

that mysterious issue where it stops signing. happened today.

uptime has been over 50 days

Nov 29 07:13:31 hsm tmkms[10685]: Nov 29 07:13:31.855  INFO tmkms::session: [columbus-4@tcp://a.b.c.d:port] signed PreVote:44FB64C387 at h/r/s 767557/0/1 (127 ms)
Nov 29 07:13:32 hsm tmkms[10685]: Nov 29 07:13:32.257  INFO tmkms::session: [columbus-4@tcp://a.b.c.d:port] signed PreCommit:44FB64C387 at h/r/s 767557/0/2 (128 ms)
Nov 29 07:13:36 hsm tmkms[10685]: Nov 29 07:13:36.524  INFO tmkms::session: [cosmoshub-3@tcp://a.b.c.e:port] signed PreVote:6493ED8E5A at h/r/s 4249554/0/1 (126 ms)
Nov 29 07:13:36 hsm tmkms[10685]: Nov 29 07:13:36.996  INFO tmkms::session: [cosmoshub-3@tcp://a.b.c.e:port] signed PreCommit:6493ED8E5A at h/r/s 4249554/0/2 (127 ms)
Nov 29 07:25:41 hsm systemd[1]: Stopping TMKMS...
Nov 29 07:25:41 hsm systemd[1]: Stopped TMKMS.
Nov 29 07:25:41 hsm systemd[1]: Started TMKMS.
Nov 29 07:25:41 hsm tmkms[25469]: Nov 29 07:25:41.845  INFO tmkms::commands::start: tmkms 0.9.0 starting up...

Hello.

I saw the issues already relating to what I am seeing, in issue #24 and #57. Both are resolved, so I am not sure where to go from here.

I am trying to get a softsign version of tmkms to run on the same machine as the COSMOS daemon, but am not having success.

In app.toml for the daemon, I have the following:

# TCP or UNIX socket address for Tendermint to listen on for
# connections from an external PrivValidator process
priv_validator_laddr = "tcp://127.0.0.1:46659"

For tmkms, I have the following config file:

# Tendermint KMS configuration file

## Chain Configuration

### Cosmos Hub Network

[[chain]]
id = "murarka"
key_format = { type = "bech32", account_key_prefix = "bluzellepub", consensus_key_prefix = "bluzellevalconspub" }
state_file = "/home/ubuntu/.tmkms/state/murarka-consensus.json"

## Signing Provider Configuration

### Software-based Signer Configuration

[[providers.softsign]]
chain_ids = ["murarka"]
key_type = "consensus"
path = "/home/ubuntu/.tmkms/secrets/bluzelle-consensus.key"

## Validator Configuration

[[validator]]
chain_id = "murarka"
addr = "tcp://127.0.0.1:46659"
secret_key = "/home/ubuntu/.tmkms/secrets/kms-identity.key"
protocol_version = "legacy"
reconnect = true

I used the "tmkms softsign import " command to import the private key from the validator.

Also, I do not have any idea how to configure the "[[providers.softsign]]" section. I could not find documentation anywhere. So maybe that is the issue. No idea.

Now, I am stuck.

I was following instructions from: https://medium.com/node-a-team/kms-yubihsm-set-up-l-cosmos-hub-c4a83ffbecd3

Of course, I am doing this with softsign, not YubiHSM, as I want to experiment with and understand tmkms first.

I am getting errors with both tmkms and blzd (my application's daemon).

When I run tmkms:

tmkms start -c ~/.tmkms/tmkms.toml -v
Oct 07 06:22:55.819 DEBUG abscissa_core::component::registry: registered component: abscissa_core::terminal::component::Terminal (v0.5.2)
Oct 07 06:22:55.819 DEBUG abscissa_core::component::registry: registered component: abscissa_core::trace::component::Tracing (v0.5.2)
Oct 07 06:22:55.819  INFO tmkms::commands::start: tmkms 0.8.0 starting up...
Oct 07 06:22:55.819  INFO tmkms::keyring: [keyring:softsign] added consensus Ed25519 key: bluzellevalconspub1zcjduepqns4mmd688u6z9uf7sggvf2l9k75cey8sg4mjc4gfyjsmczjscrvqd8w35h
Oct 07 06:22:55.819 DEBUG tmkms::client: registering chain: murarka
Oct 07 06:22:55.819 DEBUG tmkms::commands::start: Main thread waiting on clients...
Oct 07 06:22:55.819 DEBUG tmkms::session: [murarka@tcp://127.0.0.1:46659] connecting to validator...
Oct 07 06:22:55.819  INFO tmkms::connection::tcp: KMS node ID: 238FC854895FC41FFF68F109DE5E01518317847E
Oct 07 06:22:55.820 ERROR tmkms::client: [murarka@tcp://127.0.0.1:46659] I/O error: Connection refused (os error 111)
Oct 07 06:22:56.820 DEBUG tmkms::session: [murarka@tcp://127.0.0.1:46659] connecting to validator...
Oct 07 06:22:56.820  INFO tmkms::connection::tcp: KMS node ID: 238FC854895FC41FFF68F109DE5E01518317847E
Oct 07 06:22:56.820 ERROR tmkms::client: [murarka@tcp://127.0.0.1:46659] I/O error: Connection refused (os error 111)
Oct 07 06:22:57.820 DEBUG tmkms::session: [murarka@tcp://127.0.0.1:46659] connecting to validator...
Oct 07 06:22:57.820  INFO tmkms::connection::tcp: KMS node ID: 238FC854895FC41FFF68F109DE5E01518317847E
Oct 07 06:22:57.821 ERROR tmkms::client: [murarka@tcp://127.0.0.1:46659] cryptographic error
Oct 07 06:22:58.822 DEBUG tmkms::session: [murarka@tcp://127.0.0.1:46659] connecting to validator...
Oct 07 06:22:58.822  INFO tmkms::connection::tcp: KMS node ID: 238FC854895FC41FFF68F109DE5E01518317847E
Oct 07 06:23:00.494 ERROR tmkms::client: [murarka@tcp://127.0.0.1:46659] I/O error: Connection reset by peer (os error 104)
Oct 07 06:23:01.494 DEBUG tmkms::session: [murarka@tcp://127.0.0.1:46659] connecting to validator...
Oct 07 06:23:01.494  INFO tmkms::connection::tcp: KMS node ID: 238FC854895FC41FFF68F109DE5E01518317847E
Oct 07 06:23:01.495 ERROR tmkms::client: [murarka@tcp://127.0.0.1:46659] I/O error: Connection refused (os error 111)
Oct 07 06:23:02.495 DEBUG tmkms::session: [murarka@tcp://127.0.0.1:46659] connecting to validator...
Oct 07 06:23:02.495  INFO tmkms::connection::tcp: KMS node ID: 238FC854895FC41FFF68F109DE5E01518317847E
Oct 07 06:23:02.495 ERROR tmkms::client: [murarka@tcp://127.0.0.1:46659] I/O error: Connection refused (os error 111)
Oct 07 06:23:03.495 DEBUG tmkms::session: [murarka@tcp://127.0.0.1:46659] connecting to validator...
Oct 07 06:23:03.495  INFO tmkms::connection::tcp: KMS node ID: 238FC854895FC41FFF68F109DE5E01518317847E
Oct 07 06:23:03.496 ERROR tmkms::client: [murarka@tcp://127.0.0.1:46659] I/O error: Connection refused (os error 111)

When I run blzd (with log_level info):

blzd start --log_level info
I[2020-10-07|06:22:57.456] starting ABCI with Tendermint                module=main 
I[2020-10-07|06:22:57.466] Module setup                                 module=main bluzelle_crud=true
I[2020-10-07|06:22:57.486] Starting multiAppConn service                module=proxy impl=multiAppConn
I[2020-10-07|06:22:57.486] Starting localClient service                 module=abci-client connection=query impl=localClient
I[2020-10-07|06:22:57.486] Starting localClient service                 module=abci-client connection=mempool impl=localClient
I[2020-10-07|06:22:57.486] Starting localClient service                 module=abci-client connection=consensus impl=localClient
I[2020-10-07|06:22:57.486] Starting EventBus service                    module=events impl=EventBus
I[2020-10-07|06:22:57.486] Starting PubSub service                      module=pubsub impl=PubSub
I[2020-10-07|06:22:57.492] Starting IndexerService service              module=txindex impl=IndexerService
I[2020-10-07|06:22:57.492] ABCI Handshake App Info                      module=consensus height=34 hash=8A0BD536FFFEA92EE78EF2574568C03BEBF4606F55D2E0D38B09126BD42B8771 software-version= protocol-version=0
I[2020-10-07|06:22:57.492] ABCI Replay Blocks                           module=consensus appHeight=34 storeHeight=34 stateHeight=34
I[2020-10-07|06:22:57.492] Completed ABCI Handshake - Tendermint and App are synced module=consensus appHeight=34 appHash=8A0BD536FFFEA92EE78EF2574568C03BEBF4606F55D2E0D38B09126BD42B8771
I[2020-10-07|06:22:57.492] Starting SignerListenerEndpoint service      module=privval impl=SignerListenerEndpoint
I[2020-10-07|06:22:57.492] SignerListener: Listening for new connection module=privval 
ERROR: error with private validator socket client: can't get pubkey: endpoint connection timed out

I have no idea at this point. I tried to search everywhere but cannot figure out the issue. Is there some handshake that I need to configure (wild guess)? I don't have any documentation on this.

Please advise. Thanks.

This is a tracking issue for potentially adding support for Ethereum transactions.

We are considering using the ethers.rs library to produce Ethereum transactions in a similar manner to how we use the stdtx crate to produce Tendermint transactions.

Ideally we can support both YubiHSM2 and Soft Sign backends for this purpose, and potentially Ledger as well.

Summary

Tendermint 0.35 is a couple of months out, but I would like to get some testing in with gRPC.

• gRPC Support
• gRPC as an alternative to the current custom connection was integrated and tested. tmkms should begin writing support for this as well. I can take a stab at this, I am fairly inexperienced with rust, but it could be a good thing to play around with if no one wants to pick this up.

in reference to ->
https://github.com/iqlusioninc/tmkms/blob/develop/src/commands/init/templates/schema/cosmos-sdk.toml#L1-L4

Was looking for tmkms signing implementation for cosmos-sdk types, didn't find any example.
it would be great to have an example, also so that we could expand the library.

Tendermint 0.34 is expected to be backwards compatible in the block signing proposal.

It is possible to drop amino entirely and use protobuf for Tendermint 0.34 signing and we should but the bytes on the wire should be identical. We should validate this week.

@tony-iqlusion @Shella

Can we get a changelog for the v0.10 please? I know it's not released yet but would still be helpful for folks looking at the beta, and namely answering the question of what version of Tendermint it's compatible with.

Thanks!

tendermint/tendermint#4698

This issue is a discussion/research issue to track the required items for Tendermint 0.34 compatibility. There are other issues open that relate to this, but at one point all of them need to be implemented to release a compatible KMS. (I don't have the permissions to create a milestone.)

Please add any new identified ideas or issues here.

• Dependency upgrades: signatory v0.21 v0.22 (this depends on an stdtx dependency upgrade which will hopefully take place with the above protobuf upgrade)
• #97 - gRPC Transaction Sources (also mentioned in #96 tx-signer: post-Stargate roadmap)
• #73 gRPC support (possible overlap with #97 but the point is there are some gRPC stuff to do)
• Upgrade to Tendermint-rs v0.17 domain types (this adds protobuf encoding)
• Upgrade KMS amino types to protobuf types
• #212 ledger-validator-app support intent to deprecate, see #292

tmkms service stops signing and the validator is replying with:
E[2020-11-10|20:54:32.366] SignerListener: Ping timeout module=privval

There is no error msg on the tmkms side, only the validator is reporting this error. I need to restart both validator / tmkms in order to fix the issue.

This is happening on multiple networks (iris/terra/kava/gaiad) with version tmkms 0.8.0 and we have this issue starting with version 0.6.0.

Log from today from terra validator:

Nov 10 20:48:50 terrad[190405]: I[2020-11-10|20:48:50.547] Executed block                               module=state height=513449 validTxs=1 invalidTxs=0
Nov 10 20:48:50 terrad[190405]: I[2020-11-10|20:48:50.572] Committed state                              module=state height=513449 txs=1 appHash=A052FB0C265ABC247168289BE647747F848DA1D4DB8EBEDD6D7892505A7DB991
Nov 10 20:48:55 tmkms[190307]: #033[2mNov 10 20:48:55.350 #033[0m#033[32m INFO#033[0m tmkms::session: [columbus-4@tcp://127.0.0.1:26658] signed PreVote:79E334B43D at h/r/s 513450/0/1 (126 ms)
Nov 10 20:48:55 tmkms[190307]: #033[2mNov 10 20:48:55.689 #033[0m#033[32m INFO#033[0m tmkms::session: [columbus-4@tcp://127.0.0.1:26658] signed PreCommit:79E334B43D at h/r/s 513450/0/2 (125 ms)
Nov 10 20:48:56 terrad[190405]: I[2020-11-10|20:48:56.583] Executed block                               module=state height=513450 validTxs=0 invalidTxs=0
Nov 10 20:48:56 terrad[190405]: I[2020-11-10|20:48:56.669] Committed state                              module=state height=513450 txs=0 appHash=64419CB1866D5BE0296737F7784E7820100F4B76B9F86E92731832822852A95F
Nov 10 20:49:02 tmkms[190307]: #033[2mNov 10 20:49:02.100 #033[0m#033[32m INFO#033[0m tmkms::session: [columbus-4@tcp://127.0.0.1:26658] signed PreVote:CC5B90AC48 at h/r/s 513451/0/1 (126 ms)
Nov 10 20:49:08 terrad[190405]: E[2020-11-10|20:49:08.348] SignerListener: Ping timeout                 module=privval
Nov 10 20:49:11 terrad[190405]: E[2020-11-10|20:49:11.349] SignerListener: Ping timeout                 module=privval

Thank you

After Cosmos 0.43, pubkeys are now formatted as json instead of bech32 like

{"@type":"/cosmos.crypto.ed25519.PubKey","key":"sEEsVGkXvyewKLWMJbHVDRkBoerW0IIwmj1rHkabtHU="}

failure is officially deprecated/unmaintained

The failure crate is officially end-of-life: it has been marked as deprecated
by the former maintainer, who has announced that there will be no updates or
maintenance work on it going forward.

The following are some suggested actively developed alternatives to switch to:

• anyhow
• eyre
• fehler
• snafu
• thiserror

See advisory page for additional details.

HTTP Request smuggling through malformed Transfer Encoding headers

HTTP pipelining issues and request smuggling attacks are possible due to incorrect
Transfer encoding header parsing.

It is possible conduct HTTP request smuggling attacks (CL:TE/TE:TE) by sending invalid Transfer Encoding headers.

By manipulating the HTTP response the attacker could poison a web-cache, perform an XSS attack, or obtain sensitive information
from requests other than their own.

See advisory page for additional details.

tmkms version: 0.10.0-alpha8
tendermint version: v0.34
os: Ubuntu 20.04

What happens?

I have set up a validator and a KMS in the different machine but it fails with some error both on validator & tmkms

On tmkms:

DEBUG tmkms::session: [testing@tcp://10.2.0.2:12345] connecting to validator...

tmkms::connection::tcp: KMS node ID: 5159D9ECDC9F3B1D19BE9DE0405359ECF0C2A7CC

Dec 13 13:22:52.059 INFO tmkms::session: [testing@tcp://10.2.0.2:12345] connected to validator successfully

Dec 13 13:22:52.059 WARN tmkms::session: [testing@tcp://10.2.0.2:12345]: unverified validator peer ID! (2BCB53B13E886F135736B0B916F15A537B0A7891)

Dec 13 13:22:52.060 ERROR tmkms::client: [testing@tcp://10.2.0.2:12345] protocol error: invalid RPC message: None

Dec 13 13:22:53.060 INFO tmkms::connection::tcp: KMS node ID: 39AB51CD904C0785F81EE21CC8E464A932155997

Dec 13 13:22:53.062 INFO tmkms::session: [testing@tcp://10.2.0.2:12345] connected to validator successfully

Dec 13 13:22:53.062 WARN tmkms::session: [testing@tcp://10.2.0.2:12345]: unverified validator peer ID! (2BCB53B13E886F135736B0B916F15A537B0A7891)

Dec 13 13:22:53.062 ERROR tmkms::client: [testing@tcp://10.2.0.2:12345] protocol error: invalid RPC message: None

Dec 13 13:22:54.062 INFO tmkms::connection::tcp: KMS node ID: 39AB51CD904C0785F81EE21CC8E464A932155997

Dec 13 13:22:54.063 INFO tmkms::session: [testing@tcp://10.2.0.2:12345] connected to validator successfully

Dec 13 13:22:54.063 WARN tmkms::session: [testing@tcp://10.2.0.2:12345]: unverified validator peer ID! (2BCB53B13E886F135736B0B916F15A537B0A7891)

Dec 13 13:22:54.063 ERROR tmkms::client: [testing@tcp://10.2.0.2:12345] protocol error: invalid RPC message: None

On validator:

SignerListener: Ping timeout module=privval
SignerListener: Ping timeout module=privval
SignerListener: Ping timeout module=privval

My configuration

On tmkms:

[[chain]]
id = "testing"
key_format = { type = "bech32", account_key_prefix = "cosmospub", consensus_key_prefix = "cosmosvalconspub" }

## Validator configuration
[[validator]]
addr = "tcp://10.2.0.2:12345"
chain_id = "testing"
reconnect = true
secret_key = "/root/.tmkms/secrets/kms-identity.key"
protocol_version = "v0.34"

## Signing provider configuration
[[providers.softsign]]
chain_ids = ["testing"]
key_type = "consensus"
path = "/root/.tmkms/signing.key"

On validator:

# priv_validator_key_file = "config/priv_validator_key.json"
...
priv_validator_laddr = "tcp://0.0.0.0:12345"

KMS run script

tmkms init /root/.tmkms

# tmkms softsign keygen /root/.tmkms/secret_connection.key

cd /root/.tmkms/ && ls
tmkms softsign import ${SELF_KMS_RELEASE}/priv_validator_key.json /root/.tmkms/signing.key

sleep 10 && tmkms start -c ${SELF_KMS_RELEASE}/tmkms.toml

I get priv_validator_key.json from the validator and put it into tmkms to generate signing.key

Please help me how to resolve this issue. (I can provide the standalone docker script for this issue)
Thanks

I am trying to configure the system to use YubiHSM. The tmkms.toml file require the path for "secret_key" and almost all online tutorial says you can get the key by using "tmkms keygen secret_connection.key" command. But "keygen" doesn't exist anymore. Can anyone tell me do I still need the "secret_key" and how can I get it If it's required?

Thanks!

Is state file format supposed to be different from validator's ?

• tmkms:

{
  "height": "222222",
  "round": "2",
  "step": 2,
  "block_id": {
    "hash": "SOMEHASH",
    "part_set_header": {
      "total": 3,
      "hash": "SOMEHASH"
    }
  }
}

• cosmos-sdk v0.42.1 app:

{
  "height": "2222222",
  "round": 2,
  "step": 2,
  "signature": "SOMESIGNATURE==",
  "signbytes": "SIGNBYTES"
}

If so how on could convert formats between each other automaticaly ?

I'm having an issue with silent failures. tmkms will stop signing, with no errors logged. It just stops signing. The last log entry is the last signature. Restarting tmkms resolves the problem.

This is v0.7.2, on ubuntu. I have this issue on two instances, with 2 keys and 3 keys on a yubihsm in use respectively. The problem re-occurs irregularly, seems to be roughly every few weeks.

At this point I have very little to go on, and no useful information to share.

Is there anything I can do to produce useful debugging information next time the failure happens?

Documentation here: https://docs.tendermint.com/master/tools/remote-signer-validation.html

This was broken in the migration from CircleCI to GitHub actions. See comment here:

https://github.com/iqlusioninc/tmkms/blob/6070a5d/.github/workflows/rust.yml#L262

spin is no longer actively maintained

The author of the spin crate does not have time or interest to maintain it.

Consider the following alternatives (both of which support no_std):

• conquer-once
• lock_api (a subproject of parking_lot)

See advisory page for additional details.