Code Monkey home page Code Monkey logo

juno's Introduction

Juno

Build Status Go Report Card

Juno is a Cosmos Hub blockchain data aggregator and exporter that provides the ability for developers and clients to query for indexed chain data.

Table of Contents

Background

Juno is a Cosmos Hub data aggregator and exporter. In other words, it can be seen as an ETL layer atop of the Cosmos Hub. It improves the Hub's data accessibility by providing an indexed PostgreSQL database exposing aggregated resources and models such as blocks, validators, pre-commits, transactions, and various aspects of the governance module. Juno is meant to run with a GraphQL layer on top so that it even further eases the ability for developers and downstream clients to answer queries such as "what is the average gas cost of a block?" while also allowing them to compose more aggregate and complex queries.

๐ŸŽ‰ Fission Labs publishes a public GraphQL API for the Cosmos Hub that can be found here ๐ŸŽ‰

Versions

  • v0.0.5 - v0.0.6
    • Network: cosmoshub-3
  • v0.0.4
    • Network: cosmoshub-2
  • v0.0.1 - v0.0.3:
    • Network: cosmoshub-1

Install

Juno takes a simple configuration. It needs to only know about a PostgreSQL database instance and a Tendermint RPC node.

Example:

rpc_node = "<rpc-ip/host>:<rpc-port>"
client_node = "<client-ip/host>:<client-port>"

[database]
host = "<db-host>"
port = <db-port>
name = "<db-name>"
user = "<db-user>"
password = "<db-password>"
ssl_mode = "<ssl-mode>"

To install the binary run make install.

Note: Requires Go 1.13+

Usage

Juno internally runs a single worker that consumes from a single queue. The queue contains block heights to aggregate and export to a PostgreSQL database. Juno will start a new block even listener where for each new block, it will enqueue the height. A worker listens for new heights and queries for various data related to the block height to persist. For each block height, juno will persist the block, the validators that committed/signed the block, all the pre-commits for the block and the transactions in the block.

In addition, it will also sync missing blocks from --start-height to the latest known height.

$ juno /path/to/config.toml [flags]

Schemas

The schema definitions are contained in the schema/ directory. Note, these schemas are not necessarily optimal and are subject to change! However, feel free to fork this tool and expand upon the schemas as you see fit. Any tweaks will most likely require adjustments to the database wrapper.

Future Improvements

  • Unit and integration tests
  • Persist governance proposals and tallies
  • Improve the db migration process
  • Implement better retry logic on failed queries
  • Implement a docker-compose setup that allows for complete bootstrapping
  • Improve modularity and remove any assumptions about the origin chain so Juno can sync with any Tendermint-based chain

Contributing

Contributions are welcome! Please open an Issues or Pull Request for any changes.

License

MPL 2.0

juno's People

Contributors

alexanderbez avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

juno's Issues

Implement better retry logic on failed queries

Currently the worker naively retries every failed export:

// start starts a worker by listening for new jobs (block heights) from the
// given worker queue. Any failed job is logged and re-enqueued.
func (w worker) start() {
	for i := range w.queue {
		if err := w.process(i); err != nil {
			// re-enqueue any failed job
			// TODO: Implement exponential backoff or max retries for a block height.
			go func() {
				log.Printf("re-enqueueing failed block %d\n", i)
				w.queue <- i
			}()
		}
	}
}

A better approach is needed to not continuously retry the same heights.

Validator may not be known when block is persisted

In worker#process we first export and persist all precommits/signature in the block found in LastCommit and then we export and persist the block itself. There is a foreign key in block (proposer_address) that maps to the validators table. However, the block's proposer may not exist in the validator table if the proposer is new and has never signed before.

Improve modularity

Improve modularity and remove any assumptions about the origin chain so Juno can sync with any Tendermint-based chain.

Connectivity issues with LCD/RPC

I'm experimenting with cosmos, and I'm having some issue connecting to the RPC and LCD endpoints.

If I curl both LCD and RPC endpoints I get a reply from the network, but when I use the following config.toml to connect to the network I get the following error

{"level":"info","error":"Validators: response error: RPC error -32603 - Internal error: height must be greater than 0","height":83,"time":"2020-02-27T17:07:48Z","message":"failed to get validators for block"}

config.toml
rpc_node = "http://127.0.0.1:26657"
client_node = "http://127.0.0.1:1317"

[database]
host = "localhost"
port = 5432
name = "postgredb"
user = "admin"
password = "password"
ssl_mode = "disable"

I have setup multiple validators, and blocks are being produced normally.

Is there something wrong in my configuration?

Thanks a lot

Improve queue consumer/producer model

Improve queue consumer/producer model to perhaps improve speed of syncing. Currently we have a single producer/consumer model. Although, this mostly impacts syncing from an already large existing chain.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.