Code Monkey home page Code Monkey logo

ethereum-indexer's Introduction

Indexing Anywhere

A modular indexer system for ethereum and other blockchain following the same RPC standard.

Git Repo: https://github.com/jolly-roger-eth/ethereum-indexer

You can find some demoes in the examples folder

And here is the Documentation Website

Main features:

  • written in typescript, run both in a browser context and node
  • modular : you can use the part you want
  • designed to run in-browser and relies only on EIP-1193
  • when run on a server, you can hook your own database module to store the indexer process's result
  • A json object can be used as DB (useful for in-browser indexing).
  • Supports Reorg
  • Supports caching

Why ?

The main reason for building ethereum-indexer is to have the indexing be performed in a fully decentralised manner: in the client.

This obviously does not scale for all use-case: try indexing all ERC20/ERC721 and the amount of log to fetch is too big to be useful, in a browser context.

But for some use case it is actually possible and efficient. This is the case where the amount of event is bounded or its scale rate is limited.

It is for example possible to instead of indexing all ERC721, to simply index the ERC721 of the current account.

Caveats

Due to the limitation of EIP-1193 (no batch request) and the current JSON RPC spec (no timestamp available in eth_getLogs result (See improvement proposal's discussion)) the indexer processors are expected to not make use of these features.

Using these features would work in a server environment where results can be cached across load-balanced instanced, but in a browser environment where each user would have its own instance, these would slow down the indexing too much.

Having said that an hybrid approach is possible where a server index and the in-browser indexer exists only as a backup when every server instances are unavailable expect for a cache (which could even be shared across user in p2p manner).

It is also worth noting that for an indexer to work, it needs to index all events and depending on the games or applications, this might not fit in memory or in browser storage qutoa. For such case, there is no other option to have that handled by a remote service.

Usage

install ethereum-indexer-browser and ethereum-indexer-js-processor

npm i ethereum-indexer-browser ethereum-indexer-js-processor

If you use react, here is a mostly self-contained example from App.tsx

import './App.css';
import {fromJSProcessor, JSProcessor} from 'ethereum-indexer-js-processor';
import {createIndexerState, keepStateOnIndexedDB} from 'ethereum-indexer-browser';
import {connect} from './utils/web3';
import react from 'react';

// we need the contract info
// the abi will be used by the processor to have its type generated, allowing you to get type-safety
// the adress will be given to the indexer, so it index only this contract
// the startBlock field allow to tell the indexer to start indexing from that point only
// here it is the block at which the contract was deployed
const contract = {
	abi: [
		{
			anonymous: false,
			inputs: [
				{
					indexed: true,
					name: 'user',
					type: 'address',
				},
				{
					indexed: false,
					name: 'message',
					type: 'string',
				},
			],
			name: 'MessageChanged',
			type: 'event',
		},
	],
	address: '0x21d366ee3BbF67AB057c517380D37E54fFd9dfC0',
	startBlock: 3040661,
} as const;

// we define the type of the state computed by the processor
// we can also declare it inline in the generic type of JSProcessor
type State = {greetings: {account: `0x${string}`; message: string}[]};

// the processor is given the type of the ABI as Generic type to get generated
// it also specify the type which represent the current state
const processor: JSProcessor<typeof contract.abi, State> = {
	// you can set a version, ideally you would generate it so that it changes for each change
	// when a version changes, the indexer will detect that and clear the state
	// if it has the event stream cached, it will repopulate the state automatically
	version: '1.0.1',
	// this function set the starting state
	// this allow the app to always have access to a state, no undefined needed
	construct() {
		return {greetings: []};
	},
	// each event has an associated on<EventName> function which is given both the current state and the typed event
	// each event's argument can be accessed via the `args` field
	// it then modify the state as it wishes
	// behind the scene, the JSProcessor will handle reorg by reverting and applying new events automatically
	onMessageChanged(state, event) {
		const greetingFound = state.greetings.find((v) => v.account === event.args.user);
		if (greetingFound) {
			greetingFound.message = event.args.message;
		} else {
			state.greetings.push({
				message: event.args.message,
				account: event.args.user,
			});
		}
	},
};

// we setup the indexer via a call to `createIndexerState`
// this setup a set of observable (subscribe pattern)
// including one for the current state (computed by the processor above)
// and one for the syncing
// we then call `.withHooks(react)` to transform these observable in react hooks ready to be used.
const {init, useState, useSyncing, startAutoIndexing} = createIndexerState(fromJSProcessor(processor)(), {
	keepState: keepStateOnIndexedDB('basic') as any,
}).withHooks(react);

// we now need to get a handle on a ethereum provider
// for this app we are simply using window.ethereum
const ethereum = (window as any).ethereum;

// but to not trigger a metamask popup right away we wrap that in a function to be called via a click of a button
function start() {
	if (ethereum) {
		// here we first connect it to the chain of our choice and then initialise the indexer
		// see ./utils/web3
		connect(ethereum, {
			chain: {
				chainId: '11155111',
				chainName: 'Sepolia',
				rpcUrls: ['https://rpc.sepolia.org'],
				nativeCurrency: {name: 'Sepolia Ether', symbol: 'SEP', decimals: 18},
				blockExplorerUrls: ['https://sepolia.etherscan.io'],
			},
		}).then(({ethereum}) => {
			// we already setup the processor
			// now we need to initialise the indexer with
			// - an EIP-1193 provider (window.ethereum here)
			// - source config which includes the chainId and the list of contracts (abi,address. startBlock)
			init({
				provider: ethereum,
				source: {chainId: '11155111', contracts: [contract]},
			}).then(() => {
				// this automatically index on a timer
				// alternatively you can call `indexMore` or `indexMoreAndCatchupIfNeeded`, both available from the return value of `createIndexerState`
				// startAutoIndexing is easier but manually calling `indexMore` or `indexMoreAndCatchupIfNeeded` is better
				// this is because you can call them for every `newHeads` eth_subscribe message
				startAutoIndexing();
			});
		});
	}
}

function App() {
	// we use the hooks to get the latest state and make sure react update the values as they changes
	const $state = useState();
	const $syncing = useSyncing();

	if (!ethereum) {
		return (
			<div className="App">
				<h1>Indexing a basic example</h1>
				<p>To test this app, you need to have a ethereum wallet installed</p>
			</div>
		);
	}
	// we have various variable to check the status of the indexer
	// here we can act on whether the indexer is still waiting to be provided an EIP-1193 provider
	if ($syncing.waitingForProvider) {
		return (
			<div className="App">
				<h1>Indexing a basic example</h1>
				<button onClick={start} style={{backgroundColor: '#45ffbb', color: 'black'}}>
					Start
				</button>
			</div>
		);
	}

	// here we add a progress bar indicating the progress of the indexer
	return (
		<div className="App">
			<h1>Indexing a basic example</h1>
			<p>{$syncing.lastSync?.syncPercentage || 0}</p>
			{$syncing.lastSync ? (
				<progress value={($syncing.lastSync.syncPercentage || 0) / 100} style={{width: '100%'}} />
			) : (
				<p>Please wait...</p>
			)}
			<div>
				{$state.greetings.map((greeting) => (
					<p key={greeting.account}>{greeting.message}</p>
				))}
			</div>
		</div>
	);
}

export default App;

ethereum-indexer's People

Contributors

wighawag avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.