bitcoinjs / indexd Goto Github PK
View Code? Open in Web Editor NEWAn external bitcoind index management service module
License: ISC License
An external bitcoind index management service module
License: ISC License
Adapter.prototype.connectBlock
and Adapter.prototype.disconnectBlock
transaction iteration to be done asynchronously, allowing for inner-loop asynchronous flowlet fee = inAccum - outAccum
Work in progress branch: https://github.com/dcousens/indexd/tree/feeindex
May or may not be suitable, but it probably could work to be honest...
Provided you only communicate with a trusted node - indexd
doesn't do any blockchain validation.
Then it'd just be a matter of -addnode
to connect up rather than RPC.
a) Install and start bitcoind and indexd for the first time (both were syncing from scratch)
b) Last night SSH sessions to these systems dropped because notebook went to sleep mode. bitocoind was running in the background, so it didn't get interrupted.
c) This morning I connected again and started indexd (yesterday it was running in foreground so it stopped when SSH connection dropped). As shown above, indexd API is now working and showing the current, correct chainBlock, but blocksBehind is increasing (i.e. indexing is not working, blockIndex is stuck at 1211212)
root@indexd:/data/indexd/indexd-testnet# curl http://localhost:18432/status
{"chainBlock":1211215,"indexBlock":1211212,"blocksBehind":3,"ready":false}
indexd LOG file:
# tail -f LOG
2017/10/28-05:54:54.264780 7f6eb9cca700 Recovering log #15703
2017/10/28-05:55:11.497125 7f6eb9cca700 Level-0 table #15714: started
2017/10/28-05:55:16.870921 7f6eb9cca700 Level-0 table #15714: 475144539 bytes OK
2017/10/28-05:55:16.908894 7f6eb9cca700 Delete type=3 #9
2017/10/28-05:55:16.909430 7f6eb9cca700 Delete type=0 #15703
Bitcoin daemon status:
root@indexd:/data/indexd/indexd-testnet# bitcoin-cli getblockchaininfo
{
"chain": "test",
"blocks": 1211215,
"headers": 1211215,
"bestblockhash": "000000001eb9ec8f83980220f52bff577c06a0798ea77a6a2133cb2f1f212322",
"difficulty": 1,
"mediantime": 1509166914,
"verificationprogress": 0.9999985607027567,
"chainwork": "0000000000000000000000000000000000000000000000318e74502323fed131",
"pruned": false,
"softforks": [
{
"id": "bip34",
"version": 2,
"reject": {
"status": true
}
},
{
"id": "bip66",
"version": 3,
"reject": {
"status": true
}
},
{
"id": "bip65",
"version": 4,
"reject": {
"status": true
}
}
],
"bip9_softforks": {
"csv": {
"status": "active",
"startTime": 1456790400,
"timeout": 1493596800,
"since": 770112
},
"segwit": {
"status": "active",
"startTime": 1462060800,
"timeout": 1493596800,
"since": 834624
}
}
}
I checked to make sure that I have just one indexd running.
# ps -ef | grep index
root 8930 8920 0 05:54 ? 00:00:00 node /usr/bin/forever index.js
root 8940 8930 0 05:54 ? 00:00:24 /usr/bin/node /indexd/index.js
root 9244 8901 0 07:18 ? 00:00:00 grep --color=auto index
Is this normal?
I stopped and started indexd and it caught up. I'll now leave it running in foreground and hopefully duplicate this problem.
root@indexd:/indexd# nodejs --version
v6.11.5
root@indexd:/indexd# npm --version
3.10.10
I ran out of disk space while building index, which partially corrupted my Bitcoin data.
It'd be nice if indexd had a "low disk space" setting so that we could have it auto-shutdown before it gets a chance to do damage.
I see a ton of
zmq 866 messages lost +133ms
zmq 227 messages lost +1s
zmq 534 messages lost +152ms
zmq 205 messages lost +128ms
zmq 523 messages lost +7ms
zmq 976 messages lost +136ms
zmq 996 messages lost +329ms
zmq 425 messages lost +495ms
zmq 721 messages lost +4ms
zmq 1015 messages lost +147ms
zmq 486 messages lost +146ms
zmq 157 messages lost +294ms
zmq 1072 messages lost +478ms
zmq 624 messages lost +4ms
zmq 1178 messages lost +141ms
zmq 818 messages lost +145ms
zmq 824 messages lost +340ms
zmq 301 messages lost +166ms
zmq 231 messages lost +139ms
zmq 739 messages lost +3ms
zmq 856 messages lost +126ms
When its sycning up, I assume this is handled and is just for debug purposes or are the indexes incomplete if this appears?
Is there any code or documentation around block reorgs and how indexd handles them? I'm referring to the types of block reorgs illustrated in this technical explanation.
I'm just starting to look through the code and get the indexer running. If there are specific places in the code that handle re-orgs, please drop some links.
@Runn1ng I assume this is what you mean by timestamp
index?
I'll push up this, as its only in my local fork of indexd
atm...
With orphans retained to enable non-SPV clients to roll-back if required
If a system failure occurs during writing of the 2nd order data (e.g out of memory/space), then upon resume the 2nd order data is skipped due to the tip
update not rolling back.
Hello, my bitcoin node (v0.16.99.0-e057589) crashes without throwing an error while syncing indexd. It happens after a few hours and is really frustrating. I'm using the latest release in this server but I figured it probably is an indexd issue: https://github.com/CounterpartyXCP/indexd-server
bitcoin.conf
listen=1
rpcuser=xxxxxxxxxxxxx
rpcpassword=xxxxxxxxxxxx
rpcport=8332
rpcconnect=127.0.0.1
rpcallowip=222.22.22.22
rpcallowip=127.0.0.1
rpcallowip=22.222.222.22
zmqpubhashtx=tcp://127.0.0.1:38832
zmqpubhashblock=tcp://127.0.0.1:38832
server=1
txindex=1
addrindex=1
walletbroadcast=0
daemon=1
rpcthreads=16
rpcworkqueue=32
Can someone created a simple Getting Started section in README?
I assume I need to install NodeJS, NPM, then this, and create a custom https://github.com/bitcoinjs/indexd/blob/master/test/index.js-like config file? Does that config file replace the main index.js file, or is it supposed to exist in a separate directory like here?
How to execute?
How to query the index?
I noticed this negative number of lost messages. Maybe the formula isn't completely correct.
root@indexd:/indexd# forever index.js
warn: --minUptime not set. Defaulting to: 1000ms
warn: --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms
2017-10-28T12:08:15.260Z index Initializing blockchain connection (for testnet)
2017-10-28T12:08:15.350Z index starting API server
2017-10-28T12:08:15.357Z index App listening on port 18432
2017-10-28T12:23:43.160Z zmq -494 messages lost
2017-10-28T12:23:43.451Z zmq 493 messages lost
Partially resolved by 46361a4
Unfortunately, that query doesn't return the maximum height
achieved so you can re-continue the query later without starting from 0
.
That should be added somehow.
For now, this prevents DoS.
Tested with https://blockchain.info/popular-addresses
I did some testing and found I get slower, but still acceptable performance with very minimalist settings: just 1 RPC concurrent and only 32 RPC requests per batch (compared to 32 and 500, respectively, in https://github.com/bitcoinjs/indexd/blob/v0.8.2/example/.env#L5).
These tests from the table were done as follows:
a) Start bitcoind on testnet and let it catch up with the network
b) Delete indexd data, start indexd and stop it when it catches up
I used bitcoin 0.15.1rc1 and indexd 0.8.2. This system has plenty of resources and uses SSD.
In prior days (not on the chart) using the default indexd .env
settings with bitcoind adjusted to 32 RPC threads and queue of 500, I'd get (address) index built in as little as 70 minutes, but with a lot more dropped messages and (IMO) unnecessarily high stress on bitcoind.
With my minimalist settings I also got the lowest number of missed ZMQ messages. Your mileage may vary (I didn't have any application (from indexd clients) workload) but IMO indexd is an important, but not critical service, and my lowest settings work just fine while not placing bitcoind at risk (while we know that aggressive settings can kill bitcoind).
I would suggest to change the defaults to values that aren't higher than the default related bitcoind settings, so that those do not have to be changed (so, not more than 4 RPC threads on bitcoind). I'm not sure how batch sizes are sized, but if bitcoind's default rpcworkqueue is 16, maybe RPCBATCHSIZE should be 16 as well (maybe even 8 or 4 would work well - because the initial values were so high, I spent a lot of time until I reached low values from the chart, so I haven't had a chance to try even lower values).
Thoughts @fanatid? @jprichardson
This is an alternative to insight or the like, in that it uses the mainline https://github.com/bitcoin/bitcoin RPC and software, not a modified version.
It still uses the txindex
in bitcoin-core
(as moving that out would duplicate a lot of data), and we can either use the soon-to-be-added unspents index as in bitcoin/bitcoin#9806, or we can trivially add our own (probably ideal tbh).
... to a 3rd party module.
Note that a .Value
type can be used now instead of the .prefix
Added in d3e6fe9
Hello,
I am working on a project that has many private keys that have balances in them. I do not want to have to go out to a public API to query for security concerns.
I see there is a way to get the unspent amounts for an address in the example. However, I am not seeing a way to connect the private key to it's address. I believe this is what I would have to do to determine how much BTC the private key is linked to.
Is the data structure in indexd able to perform this type of query against the blockchain?
Thanks in advance!
index Error: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
These are reacting to bitcoind
events... so not sure how that could happen.
What are exact steps needed to launch indexd daemon on ubuntu machine with bitcoind installed?
Probably best suited to an external module
If an attached listener service throws an exception, it should not impact other potential listeners.
Partially resolved with 22994cf, but only in terms of it does not halt resync now.
Just a fast question
Would it be possible to use decoderawtransaction
RPC call to decode transaction instead of using bitcoin.js parsing?
The only reason why I am asking this is that I would love something that's coin-agnostic, and decoderawtransaction
is in all bitcoin clones, but bitcoin.js has to be sometimes specifically ported...
I'm getting the following error using the example code when starting indexd:
/root/.bitcoin/chainstate/LOCK: Resource temporarily unavailable
I'm on centos 7, same configuration has worked for me on an ubuntu server a while back. Anyone got an idea?
Hello,
from the issues I read I found out perhaps I am not using the indexd module correctly. There is a standalone index server as I could read, is this a module for it?
In the previous commit (53506f0) I could run node test/index.js and the index leveldb would be populated with debug output visible on the console.
When using the latest commit I can do queries navigating to express example page, but I do not see the db being populated anymore.
Would like to ask your kind help in setting it up, or pointing me to a readme where it is noted how to run it.
If your tooling uses addresses... you probably want to transcode addresses to their scId
(sha256(script)):
let bitcoin = require('bitcoinjs-lib')
// ...
let tasks = {}
addrs.forEach((address) => {
let script = bitcoin.address.toOutputScript(address)
let scId = bitcoin.crypto.sha256(script).toString('hex')
tasks[address] = (next) => indexd.adapter.transactionsByScriptId(scId, height, next)
})
Then you can easily map the task results back to each address
involved...
List in README? Or not necessary?
In some cases such as this one below, certain components use a different date format. Probably something from upstream, in which case feel free to close.
2017-10-30T14:35:27.153Z zmq:tx hashtx ef108ac19e62d4b2c3c0adfb4a8dde787a41abde0 6e02bcab907980c41b9821f
Mon, 30 Oct 2017 14:35:27 GMT mempool { transactions: 284, inputs: 450, outputs: 505 }
TXID | SEEN_HEIGHT | UTC
probably
From #11
We should instead maintain two indexd chains, one for 1st order data, and the other chain, for the 2nd order data...
This would allow us to independently roll-back the chains in the event of failure.
This would also mean we can run a direct import using fast-dat-parser
and allow the fee-data to catch up.
Running indexd master (7c0c125) against testnet:
indexd:rpc getblock [ '00000000003ba174fd77250a04eba0e15134da0b4edfb9996bfedffdd235f97c',
2 ] RangeError [ERR_INVALID_OPT_VALUE]: The value "250000" is invalid for option "value"
at checkInt (buffer.js:1275:11)
at Buffer.writeUInt16LE (buffer.js:1337:7)
at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/numbers.js:27:11)
at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/varbuffer.js:22:18)
at /home/adam/indexd/node_modules/varstruct/lib/object.js:36:19
at Array.forEach (<anonymous>)
at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/object.js:33:13)
at ChainedBatch.put (/home/adam/indexd/dbwrapper.js:42:35)
at outs.forEach (/home/adam/indexd/indexes/txo.js:52:14)
at Array.forEach (<anonymous>) +43ms
service RangeError [ERR_INVALID_OPT_VALUE]: The value "250000" is invalid for option "value"
service at checkInt (buffer.js:1275:11)
service at Buffer.writeUInt16LE (buffer.js:1337:7)
service at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/numbers.js:27:11)
service at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/varbuffer.js:22:18)
service at /home/adam/indexd/node_modules/varstruct/lib/object.js:36:19
service at Array.forEach (<anonymous>)
service at Object.encode (/home/adam/indexd/node_modules/varstruct/lib/object.js:33:13)
service at ChainedBatch.put (/home/adam/indexd/dbwrapper.js:42:35)
service at outs.forEach (/home/adam/indexd/indexes/txo.js:52:14)
service at Array.forEach (<anonymous>) +1m
I think it's choking on tx 22e08a53aff6cb0680758036e09f792dac56e4813de81c9be1b4b1ae3cda0ab0 which seems to be some sort of crazy non-standard tx. I couldn't decode it with bitcoin-cli
The only "extra" information returned is value
, useful, but, could probably do better.
scId
too?
Is that even useful without a mapping?
Do we want:
scId
-> script
For reverse mapping that information?
Just hitting some of the API endpoints in private-bitcoin to see what happens and ran into this when I visited /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block:
express:router dispatching GET /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +1m
express:router query : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
express:router expressInit : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
express:router corsMiddleware : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
express:router router : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +1ms
express:router dispatching GET /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
express:router <anonymous> : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
express:router <anonymous> : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
express:router trim prefix (/1) from url /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
express:router router /1 : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
express:router dispatching GET /t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
express:router <anonymous> : /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block +0ms
api Error: Expected Object, got String "eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489"
api at typeforce (/home/adam/indexd/node_modules/typeforce/index.js:198:11)
api at LevelDOWN.get (/home/adam/indexd/dbwrapper.js:25:3)
api at TxIndex.heightBy (/home/adam/indexd/indexes/tx.js:65:6)
api at Indexd.blockIdByTransactionId (/home/adam/indexd/index.js:260:19)
api at router.get (/home/adam/private-bitcoin/routes/1.js:122:14)
api at Layer.handle [as handle_request] (/home/adam/private-bitcoin/node_modules/express/lib/router/layer.js:95:5)
api at next (/home/adam/private-bitcoin/node_modules/express/lib/router/route.js:137:13)
api at hexWare (/home/adam/private-bitcoin/routes/1.js:106:5)
api at Layer.handle [as handle_request] (/home/adam/private-bitcoin/node_modules/express/lib/router/layer.js:95:5)
api at next (/home/adam/private-bitcoin/node_modules/express/lib/router/route.js:137:13) +0ms
api GET /1/t/eb160e6c981d266a5174151dee96feed60b58e69a6b49167820f32e741a24489/block 400 Bad Request 3ms 0->98B +3ms
The RPC is the limiting factor, we could trivially de-linearize the 1st order connect
calls, provided tip
doesn't get confused.
tip
is the only data that is "UPDATE"d.
We rely on ZMQ
at the moment to trigger a resync
, but, we could additionally encourage a poll
w/ a time interval.
Otherwise, if the ZMQ socket goes down, the indexd
node will cease to synchronize at all.
Easily extended... either way, its at the example/
level, not part of the module itself, but I think everyone uses example
as the reference implementation at the moment as it is non-trivial.
This would mean we could run indexd
on a pruning node, provided it is fully synchronized to start.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.