larsyngvelundin / cointrakker Goto Github PK

View Code? Open in Web Editor NEW

0.0 1.0 0.0 619 KB

A visual tool to explore BTC transactions

License: GNU General Public License v2.0

Python 65.26% JavaScript 29.14% CSS 3.71% HTML 1.90%

cointrakker's Introduction

CoinTrakker

My Thesis Project for @Medieinstitutet Blockchain Developer

A tool to explore transactions in an interactable graph that shows the flow of transactions in a visual way.

cointrakker's People

Contributors

Watchers

cointrakker's Issues

Add mention for Sankey Library used

JS - D3 Sankey Diagram MVP

Readme - Flush out

Flush out Readme with comprehensive information about project.
Requirements.
How to use.
Live page.
Example gifs.
Time to parse, and size of DB.

DB - Update the print_all function to work properly with new DB system

DB - Function to return date of last block

Save data to SQL

Write base for db handling

RPC - Handle the remainder of transactions

Example transaction: 5c98e15fd2f37006e411d0d8fc7a7b41149678caff0ee06e2fd3bad5621b17fb

1ExF15z6YxFrNZHcnH2sYohbKBh47b1aow technically sends 49.98 to themself, but the outgoing transaction is not recorded

Set most loggers to debug

JS - Show last block and its date

DB - Change .db naming to account for case-insensitive OS

DB - Automatic detection of ID or non-ID

For get_transactions functions, automatic detection of ID.
Something like:

if(isinstance(input, int)):
    id = input
else:
    id = get_id_from_input(input)

Parse - Some blocks are incorrectly skipped

Parsed DB can be used to find these blocks, as there's no mining reward saved for the skipped blocks

DB - Return transaction data from hash

Function that takes transaction hash and return a duct with information about that transaction. (Amount, block, from, to, time?, hash)

DB - Basic functions to get hash/address from ID

JS - Fetch incoming transactions as well

Rewrite db to handle amount in Satoshis instead of BTC

Flask - Profile code to check for bottlenecks

Parse - Optimize RPC usage

With the new SQL optimization, time to save is very low, even for blocks with thousands of transactions. But the RPC requests are taking up a considerable amount of time

2024-01-02 01:44:24.457 | INFO     | rpc_files.parse_blocks:main:25 - Block: 813873 with 2491 transactions
2024-01-02 01:47:00.990 | INFO     | rpc_files.parse_blocks:main:94 - Total time: 156.596
2024-01-02 01:47:00.991 | INFO     | rpc_files.parse_blocks:main:95 - Getting the block data:      0.064    (0.04%)
2024-01-02 01:47:00.991 | INFO     | rpc_files.parse_blocks:main:96 - Looping the hashes:          156.532  (99.96%)
2024-01-02 01:47:00.991 | INFO     | rpc_files.parse_blocks:main:97 -     - Parsing the block data:  155.115  (99.10%)
2024-01-02 01:47:00.991 | INFO     | rpc_files.parse_blocks:main:98 -     - Saving the block data:   1.397    (0.89%)

RPC - move get_address_from_pubkey to separate file

DB - Clean up code

Remove no longer used loops and statements
Remove commented code

Flask - Cache transaction data

When JS requests transactions. Give back first/biggest 5 and a remainder.
Next time JS requests, give another 5 from that cached list.

RPC - Handle addresses under 'scriptSig'

DB - Make 'stats.db'

A file that includes some general data like the latest block parsed and it's time stamp.

Readme - Public draft

JS - Move Sankey functions into separate files

JS - Add dynamic data for Sankey

JS - Check if wallet is already clicked

Create and test connection to Bitcoin RPC

CSS - Smooth animations for adding new elements

DB - Parser crashes on block 71036

Traceback (most recent call last):
  File "/mnt/md0/cointrakker/CoinTrakker/rpc_files/parse_transaction_data.py", line 21, in main
    to_address = item['scriptPubKey']['address']
KeyError: 'address'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/md0/cointrakker/CoinTrakker/parse_blocks_to_sql.py", line 24, in <module>
    main()
  File "/mnt/md0/cointrakker/CoinTrakker/parse_blocks_to_sql.py", line 21, in main
    rpc.parse_blocks(num1, num2)
  File "/mnt/md0/cointrakker/CoinTrakker/rpc.py", line 222, in parse_blocks
    rpc_files.parse_blocks.main(from_block, to_block)
  File "/mnt/md0/cointrakker/CoinTrakker/rpc_files/parse_blocks.py", line 22, in main
    transaction_data = rpc.parse_transaction_data(txhash)
  File "/mnt/md0/cointrakker/CoinTrakker/rpc.py", line 226, in parse_transaction_data
    return rpc_files.parse_transaction_data.main(transaction_hash)
  File "/mnt/md0/cointrakker/CoinTrakker/rpc_files/parse_transaction_data.py", line 34, in main
    to_address = rpc.get_address_from_pubkey(pubkey)
  File "/mnt/md0/cointrakker/CoinTrakker/rpc.py", line 199, in get_address_from_pubkey
    pubkey_bytes = bytes.fromhex(pubkey_hex)
ValueError: non-hexadecimal number found in fromhex() arg at position 0

DB - Redesign file structure

Combine addresses based on their n first characters.
Perhaps 1A1AEsrBHqe8Ap7PULKWrk2D9yrE77Rfkn and 1A1D4n7TTXST35qbweBTbWvbz4yu56uqot should both be under 1A.db or 1A1.db

RPC - Clean up code

Remove no longer used loops and statements
Remove commented code

Test SQL code against real blocks

Add code to fetch fee when parsing transactions

RPC - "Broken Pipe" intermittently during blocks with many transactions

Parse Blocks - Investigate bottle neck for optimization

DB - Redo structure for performance

Flask - Set up first basic version

DB - Update get_transactions_from for new DB

Formatting - Make everything AutoPEP8 formatted

Fix variable names

DB - Automatically move busy wallets to their own personal db files

RPC - KeyError: 'address' when trying to fetch sender

Traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/md0/cointrakker/CoinTrakker/rpc.py", line 217, in parse_blocks
    rpc_files.parse_blocks.main(from_block, to_block)
  File "/mnt/md0/cointrakker/CoinTrakker/rpc_files/parse_blocks.py", line 22, in main
    transaction_data = rpc.parse_transaction_data(txhash)
  File "/mnt/md0/cointrakker/CoinTrakker/rpc.py", line 221, in parse_transaction_data
    return rpc_files.parse_transaction_data.main(transaction_hash)
  File "/mnt/md0/cointrakker/CoinTrakker/rpc_files/parse_transaction_data.py", line 12, in main
    sender = rpc.get_sender_address(transaction_hash)
  File "/mnt/md0/cointrakker/CoinTrakker/rpc.py", line 213, in get_sender_address
    return rpc_files.get_sender_address.main(rpc_connection, transaction_hash)
  File "/mnt/md0/cointrakker/CoinTrakker/rpc_files/get_sender_address.py", line 15, in main
    sender_address = prev_tx_decoded['vout'][vout_index]['scriptPubKey']['address']
KeyError: 'address'

Logs:

2023-12-18 12:42:20.531 | INFO     | rpc_files.parse_blocks:main:10 - Block: 181
2023-12-18 12:42:20.531 | INFO     | rpc_files.parse_blocks:main:12 - Checking non-Genesis block
2023-12-18 12:42:20.531 | DEBUG    | rpc_files.get_block:main:3 - test
2023-12-18 12:42:20.531 | DEBUG    | rpc_files.parse_blocks:main:14 - {'hash': '00000000dc55860c8a29c58d45209318fa9e9dc2c1833a7226d86bc465afc6e5', 'confirmations': 821588, 'height': 181, 'version': 1, 'versionHex': '00000001', 'merkleroot': 'ed92b1db0b3e998c0a4351ee3f825fd5ac6571ce50c050b4b45df015092a6c36', 'time': 1231740133, 'mediantime': 1231735142, 'nonce': 792669465, 'bits': '1d00ffff', 'difficulty': 1, 'chainwork': '000000000000000000000000000000000000000000000000000000b600b600b6', 'nTx': 2, 'previousblockhash': '00000000b5ef0ea215becad97402ce59d1416fe554261405cda943afd2a8c8f2', 'nextblockhash': '0000000054487811fc4ff7a95be738aa5ad9320c394c482b27c0da28b227ad5d', 'strippedsize': 490, 'size': 490, 'weight': 1960, 'tx': ['8347cee4a1cb5ad1bb0d92e86e6612dbf6cfc7649c9964f210d4069b426e720a', 'a16f3ce4dd5deb92d98ef5cf8afeaf0775ebca408f708b2146c4fb42b41e14be']}
2023-12-18 12:42:20.531 | INFO     | rpc_files.parse_blocks:main:19 - Checking new transaction hash
8347cee4a1cb5ad1bb0d92e86e6612dbf6cfc7649c9964f210d4069b426e720a
2023-12-18 12:42:20.532 | DEBUG    | rpc_files.get_raw_transaction:main:3 - test
2023-12-18 12:42:20.542 | INFO     | rpc_files.parse_transaction_data:main:6 - Checking new transaction hash
8347cee4a1cb5ad1bb0d92e86e6612dbf6cfc7649c9964f210d4069b426e720a
2023-12-18 12:42:20.542 | DEBUG    | rpc_files.get_raw_transaction:main:3 - test
2023-12-18 12:42:20.543 | DEBUG    | rpc_files.decode_raw_transaction:main:3 - test
2023-12-18 12:42:20.543 | DEBUG    | rpc_files.parse_transaction_data:main:9 - Decoded: {'txid': '8347cee4a1cb5ad1bb0d92e86e6612dbf6cfc7649c9964f210d4069b426e720a', 'hash': '8347cee4a1cb5ad1bb0d92e86e6612dbf6cfc7649c9964f210d4069b426e720a', 'version': 1, 'size': 134, 'vsize': 134, 'weight': 536, 'locktime': 0, 'vin': [{'coinbase': '04ffff001d0128', 'sequence': 4294967295}], 'vout': [{'value': Decimal('50.00000000'), 'n': 0, 'scriptPubKey': {'asm': '0435f0d8366085f73906a48309728155532f24293ea59fe0b33a245c4b8d75f82c3e70804457b7f49322aa822196a7521e4931f809d7e489bccb4ff14758d170e5 OP_CHECKSIG', 'desc': 'pk(0435f0d8366085f73906a48309728155532f24293ea59fe0b33a245c4b8d75f82c3e70804457b7f49322aa822196a7521e4931f809d7e489bccb4ff14758d170e5)#4ma69dya', 'hex': '410435f0d8366085f73906a48309728155532f24293ea59fe0b33a245c4b8d75f82c3e70804457b7f49322aa822196a7521e4931f809d7e489bccb4ff14758d170e5ac', 'type': 'pubkey'}}]}
2023-12-18 12:42:20.543 | DEBUG    | rpc_files.get_raw_transaction:main:3 - test
2023-12-18 12:42:20.544 | DEBUG    | rpc_files.decode_raw_transaction:main:3 - test
2023-12-18 12:42:20.544 | DEBUG    | rpc_files.get_sender_address:main:7 - THIS IS A BLOCKREWARD
2023-12-18 12:42:20.545 | DEBUG    | rpc_files.parse_transaction_data:main:18 - {'value': Decimal('50.00000000'), 'n': 0, 'scriptPubKey': {'asm': '0435f0d8366085f73906a48309728155532f24293ea59fe0b33a245c4b8d75f82c3e70804457b7f49322aa822196a7521e4931f809d7e489bccb4ff14758d170e5 OP_CHECKSIG', 'desc': 'pk(0435f0d8366085f73906a48309728155532f24293ea59fe0b33a245c4b8d75f82c3e70804457b7f49322aa822196a7521e4931f809d7e489bccb4ff14758d170e5)#4ma69dya', 'hex': '410435f0d8366085f73906a48309728155532f24293ea59fe0b33a245c4b8d75f82c3e70804457b7f49322aa822196a7521e4931f809d7e489bccb4ff14758d170e5ac', 'type': 'pubkey'}}
2023-12-18 12:42:20.545 | DEBUG    | rpc_files.parse_transaction_data:main:25 - Item: {'value': Decimal('50.00000000'), 'n': 0, 'scriptPubKey': {'asm': '0435f0d8366085f73906a48309728155532f24293ea59fe0b33a245c4b8d75f82c3e70804457b7f49322aa822196a7521e4931f809d7e489bccb4ff14758d170e5 OP_CHECKSIG', 'desc': 'pk(0435f0d8366085f73906a48309728155532f24293ea59fe0b33a245c4b8d75f82c3e70804457b7f49322aa822196a7521e4931f809d7e489bccb4ff14758d170e5)#4ma69dya', 'hex': '410435f0d8366085f73906a48309728155532f24293ea59fe0b33a245c4b8d75f82c3e70804457b7f49322aa822196a7521e4931f809d7e489bccb4ff14758d170e5ac', 'type': 'pubkey'}}
2023-12-18 12:42:20.545 | DEBUG    | rpc_files.parse_transaction_data:main:26 - No plain address address found
2023-12-18 12:42:20.545 | INFO     | rpc_files.parse_transaction_data:main:31 - 50.00000000 rewarded to 1JSW4QekxPokWWU4hcRwrheZbZKSkFz9oc
2023-12-18 12:42:20.545 | INFO     | db_files.db_exists:main:5 - test
2023-12-18 12:42:20.545 | INFO     | db_files.save_transaction:main:8 - DB for 1JSW4QekxPokWWU4hcRwrheZbZKSkFz9oc does not exist
2023-12-18 12:42:20.545 | INFO     | db_files.create_db:main:5 - Creating db
2023-12-18 12:42:20.555 | INFO     | db_files.save_transaction:main:10 - Created DB files
2023-12-18 12:42:20.555 | INFO     | db_files.db_exists:main:5 - test
2023-12-18 12:42:20.555 | INFO     | db_files.save_address:main:5 - test
2023-12-18 12:42:20.556 | ERROR    | db_files.save_address:main:15 - Could not save Mining reward (probably already saved)
2023-12-18 12:42:20.556 | ERROR    | db_files.save_address:main:16 - UNIQUE constraint failed: addresses.address
2023-12-18 12:42:20.556 | INFO     | db_files.save_address:main:21 - sql data: <sqlite3.Cursor object at 0x7f789317af40>
2023-12-18 12:42:20.556 | INFO     | db_files.save_address:main:5 - test
2023-12-18 12:42:20.559 | INFO     | db_files.save_address:main:21 - sql data: <sqlite3.Cursor object at 0x7f789317ab40>
2023-12-18 12:42:20.559 | INFO     | db_files.save_hash:main:5 - test
2023-12-18 12:42:20.563 | INFO     | db_files.save_hash:main:21 - sql data: <sqlite3.Cursor object at 0x7f789317af40>
2023-12-18 12:42:20.568 | INFO     | db_files.save_transaction:main:30 - Added 8347cee4a1cb5ad1bb0d92e86e6612dbf6cfc7649c9964f210d4069b426e720a to outgoing/Mining reward.db
2023-12-18 12:42:20.571 | INFO     | db_files.save_transaction:main:41 - Added 8347cee4a1cb5ad1bb0d92e86e6612dbf6cfc7649c9964f210d4069b426e720a to incoming/1JSW4QekxPokWWU4hcRwrheZbZKSkFz9oc.db
2023-12-18 12:42:20.571 | INFO     | rpc_files.parse_blocks:main:19 - Checking new transaction hash
a16f3ce4dd5deb92d98ef5cf8afeaf0775ebca408f708b2146c4fb42b41e14be
2023-12-18 12:42:20.571 | DEBUG    | rpc_files.get_raw_transaction:main:3 - test
2023-12-18 12:42:20.577 | INFO     | rpc_files.parse_transaction_data:main:6 - Checking new transaction hash
a16f3ce4dd5deb92d98ef5cf8afeaf0775ebca408f708b2146c4fb42b41e14be
2023-12-18 12:42:20.577 | DEBUG    | rpc_files.get_raw_transaction:main:3 - test
2023-12-18 12:42:20.578 | DEBUG    | rpc_files.decode_raw_transaction:main:3 - test
2023-12-18 12:42:20.578 | DEBUG    | rpc_files.parse_transaction_data:main:9 - Decoded: {'txid': 'a16f3ce4dd5deb92d98ef5cf8afeaf0775ebca408f708b2146c4fb42b41e14be', 'hash': 'a16f3ce4dd5deb92d98ef5cf8afeaf0775ebca408f708b2146c4fb42b41e14be', 'version': 1, 'size': 275, 'vsize': 275, 'weight': 1100, 'locktime': 0, 'vin': [{'txid': 'f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16', 'vout': 1, 'scriptSig': {'asm': '3044022027542a94d6646c51240f23a76d33088d3dd8815b25e9ea18cac67d1171a3212e02203baf203c6e7b80ebd3e588628466ea28be572fe1aaa3f30947da4763dd3b3d2b[ALL]', 'hex': '473044022027542a94d6646c51240f23a76d33088d3dd8815b25e9ea18cac67d1171a3212e02203baf203c6e7b80ebd3e588628466ea28be572fe1aaa3f30947da4763dd3b3d2b01'}, 'sequence': 4294967295}], 'vout': [{'value': Decimal('10.00000000'), 'n': 0, 'scriptPubKey': {'asm': '04b5abd412d4341b45056d3e376cd446eca43fa871b51961330deebd84423e740daa520690e1d9e074654c59ff87b408db903649623e86f1ca5412786f61ade2bf OP_CHECKSIG', 'desc': 'pk(04b5abd412d4341b45056d3e376cd446eca43fa871b51961330deebd84423e740daa520690e1d9e074654c59ff87b408db903649623e86f1ca5412786f61ade2bf)#hdmzmm0s', 'hex': '4104b5abd412d4341b45056d3e376cd446eca43fa871b51961330deebd84423e740daa520690e1d9e074654c59ff87b408db903649623e86f1ca5412786f61ade2bfac', 'type': 'pubkey'}}, {'value': Decimal('30.00000000'), 'n': 1, 'scriptPubKey': {'asm': '0411db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5cb2e0eaddfb84ccf9744464f82e160bfa9b8b64f9d4c03f999b8643f656b412a3 OP_CHECKSIG', 'desc': 'pk(0411db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5cb2e0eaddfb84ccf9744464f82e160bfa9b8b64f9d4c03f999b8643f656b412a3)#u7qfa49l', 'hex': '410411db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5cb2e0eaddfb84ccf9744464f82e160bfa9b8b64f9d4c03f999b8643f656b412a3ac', 'type': 'pubkey'}}]}
2023-12-18 12:42:20.579 | DEBUG    | rpc_files.get_raw_transaction:main:3 - test
2023-12-18 12:42:20.579 | DEBUG    | rpc_files.decode_raw_transaction:main:3 - test
2023-12-18 12:42:20.580 | DEBUG    | rpc_files.get_raw_transaction:main:3 - test
2023-12-18 12:42:20.580 | DEBUG    | rpc_files.decode_raw_transaction:main:3 - test
2023-12-18 12:42:20.581 | DEBUG    | rpc_files.get_sender_address:main:14 - Decoded prev: 
{'txid': 'f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16', 'hash': 'f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16', 'version': 1, 'size': 275, 'vsize': 275, 'weight': 1100, 'locktime': 0, 'vin': [{'txid': '0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9', 'vout': 0, 'scriptSig': {'asm': '304402204e45e16932b8af514961a1d3a1a25fdf3f4f7732e9d624c6c61548ab5fb8cd410220181522ec8eca07de4860a4acdd12909d831cc56cbbac4622082221a8768d1d09[ALL]', 'hex': '47304402204e45e16932b8af514961a1d3a1a25fdf3f4f7732e9d624c6c61548ab5fb8cd410220181522ec8eca07de4860a4acdd12909d831cc56cbbac4622082221a8768d1d0901'}, 'sequence': 4294967295}], 'vout': [{'value': Decimal('10.00000000'), 'n': 0, 'scriptPubKey': {'asm': '04ae1a62fe09c5f51b13905f07f06b99a2f7159b2225f374cd378d71302fa28414e7aab37397f554a7df5f142c21c1b7303b8a0626f1baded5c72a704f7e6cd84c OP_CHECKSIG', 'desc': 'pk(04ae1a62fe09c5f51b13905f07f06b99a2f7159b2225f374cd378d71302fa28414e7aab37397f554a7df5f142c21c1b7303b8a0626f1baded5c72a704f7e6cd84c)#hsw9ejus', 'hex': '4104ae1a62fe09c5f51b13905f07f06b99a2f7159b2225f374cd378d71302fa28414e7aab37397f554a7df5f142c21c1b7303b8a0626f1baded5c72a704f7e6cd84cac', 'type': 'pubkey'}}, {'value': Decimal('40.00000000'), 'n': 1, 'scriptPubKey': {'asm': '0411db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5cb2e0eaddfb84ccf9744464f82e160bfa9b8b64f9d4c03f999b8643f656b412a3 OP_CHECKSIG', 'desc': 'pk(0411db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5cb2e0eaddfb84ccf9744464f82e160bfa9b8b64f9d4c03f999b8643f656b412a3)#u7qfa49l', 'hex': '410411db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5cb2e0eaddfb84ccf9744464f82e160bfa9b8b64f9d4c03f999b8643f656b412a3ac', 'type': 'pubkey'}}]}

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.