ethgasstation / ethgasstation-backend Goto Github PK

Adaptive Gas Price Oracle for Ethereum Blockchain

License: GNU General Public License v3.0

Python 99.50% Dockerfile 0.50%

ethgasstation-backend's Introduction

ETH Gas Station

EthGasStation is a tool for monitoring the time transactions take to be confirmed on the ETH network. It also monitors and estimates miner policies with respect to gas prices and formats the data for viewing over http.

Note that this UI is facing deprecation and will not be supported long-term. Many changes have been made to this UI in order to move EGS forward, and a new UI/frontend is in the works that relies on the new API layer.

You can view a working legacy implementation of ETH Gas Station at gasstation-legacy.etheria.io. Note that this is a demo version of this code, and thus should not be relied upon for production data. Use ethgasstation.info for all production data.

Requirements

This version of the ETH Gas Station frontend has multiple requirements:

PHP
Python 3 for running regressions, or if you want to use Redis
A MySQL-compatible data store
A working ETH Gas Station backend service

The frontend reads JSON files generated by the ETH Gas Station backend, which is a submodule for this project. See the backend repository for information on running the backend. Run the backend with --generate-report so this frontend will get the files it needs.

Usage

Update build/php/common.php with your own data.
Set up the backend to export data to json/ in the root directory.
Run the backend. The backend will need time to generate files from transactions.
Run the frontend and view the data.

If you are having trouble, make sure that json files are in json/ and you can view the JSON files appropriately.

Using Redis with this legacy frontend

It is possible to separate the backend and frontend, and not run all of the ETH Gas Station on a single server. In fact, this is how ETH Gas Station works in production. JSON generated by the ETH Gas Station backend is placed into Redis, and all database data is in a separate MariaDB instance on a different computer.

This legacy frontend assumes it will get JSON from flat files in the json/ directory. data_analysis/make_json.py contains a Python script that will take the JSON files generated by the backend in Redis and write them down to flat files in the directory structure. Running this script will attempt to update the JSON for the UI every 15 seconds (roughly once per block created.) You will need to update make_json.py with the info that points at your Redis cluster.

License

The ETH Gas Station frontend, like the backend, is GNU GPL 3.0. We have chosen the GPL 3.0 specifically to compel disclosure of modifications to the ETH Gas Station. We have created all of this as a community service, and licenses such as MIT give too much permission for commercial projects that have wanted to profit off of the ETH Gas Station without making their own modifications available to the Ethereum community or supporting the ETH Gas Station project.

If you are interested in commercially licensing ETH Gas Station technology for closed-source use, please reach out to ethgasstation AT protonmail {dot} com.

ethgasstation-backend's People

Contributors

Stargazers

Watchers

ethgasstation-backend's Issues

Gas Burners ID support

"Gas Burners" is a great idea to indicate the activity status of Dapps on Ethereum.

Ideally, each contract address (of the top 100 contracts) should map with its ID to make the UI more intuitive. However, it seems many of the IDs are missing, while some exists (e.g., IDEX, cryptokitties, etc). User could not tell why some IDs are there, while the others do not.

ID should be better supported for better illustration purpose. Or, should there be a way for the contract creator to provide the ID of the contract to ethgasstation?

ETH GasStation API

The oracle generates json files with the key data about gas price recommendations and confirmation times. The data from the json files should be available through a simple API available over http so that people do not have to parse the entire json files themselves. Basic documentation for the API is also needed.

Unknown column 's1hago'

Hello, I'm getting the following error:

[Errno 2] No such file or directory: '/home/fvictorio/repos/json/ethgasAPI.json'
Traceback (most recent call last):
  File "/home/fvictorio/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1159, in _execute_context
    context)
  File "/home/fvictorio/.local/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 467, in do_executemany
    cursor.executemany(statement, parameters)
  File "/home/fvictorio/.local/lib/python3.5/site-packages/mysql/connector/cursor.py", line 664, in executemany
    return self.execute(stmt)
  File "/home/fvictorio/.local/lib/python3.5/site-packages/mysql/connector/cursor.py", line 561, in execute
    self._handle_result(self._connection.cmd_query(stmt))
  File "/home/fvictorio/.local/lib/python3.5/site-packages/mysql/connector/connection.py", line 525, in cmd_query
    result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
  File "/home/fvictorio/.local/lib/python3.5/site-packages/mysql/connector/connection.py", line 427, in _handle_result
    raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1054 (42S22): Unknown column 's1hago' in 'field list'

I'm running this against a Rinkeby node, started with the following command:

geth --rinkeby --rpc --rpcapi="db,eth,net,web3,personal,txpool" console 2>/dev/null

Thanks.

ValueError: {'code': -32601, 'message': 'The method txpool_content does not exist/is not available'}

I haven't been able to get this backend to run properly.

Running on Ubuntu 18

python version

$ python3 --version
Python 3.6.7

geth version

$ geth version
WARN [04-12|16:37:20.962] Sanitizing cache to Go's GC limits       provided=1024 updated=656
Geth
Version: 1.8.26-stable
Git Commit: cdae1c59abc32f85debfa29577fbf1ed036ebf73
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.10.4
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go-1.10

Starting my geth instance with

$ geth  --rpc --datadir "/home/bang/Documents/geth-data"

The console output is the following (repeating the error)

bang@ubuntu:~/Documents/ethgasstation-backend$ ./ethgasstation.py 
      __  __                     __       __  _         
 ___ / /_/ /  ___ ____ ____ ___ / /____ _/ /_(_)__  ___ 
/ -_) __/ _ \/ _ `/ _ `(_-<(_-</ __/ _ `/ __/ / _ \/ _ \
\__/\__/_//_/\_, /\_,_/___/___/\__/\_,_/\__/_/\___/_//_/
            /___/                                       

[2019-04-12 16:30:14] [INFO]  Type ctl-c to quit and save data to mysql
[2019-04-12 16:30:14] [INFO]  blocks 0
[2019-04-12 16:30:14] [INFO]  txcount 0
[2019-04-12 16:30:14] [INFO]  getting txpool hashes at block 0 ...
[2019-04-12 16:30:14] [INFO]  Switching pending tx acquisition method to parity
[2019-04-12 16:30:14] [ERROR] Traceback (most recent call last):
  File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 122, in _get_pending_tx_hashes
    txpoolcontent = web3.txpool.content
  File "/home/bang/.local/lib/python3.6/site-packages/web3/txpool.py", line 9, in content
    return self.web3.manager.request_blocking("txpool_content", [])
  File "/home/bang/.local/lib/python3.6/site-packages/web3/manager.py", line 106, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method txpool_content does not exist/is not available'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 146, in append_current_txp
    hashlist = self._get_pending_tx_hashes()
  File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 138, in _get_pending_tx_hashes
    return self._get_pending_tx_hashes(False)
  File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 129, in _get_pending_tx_hashes
    txpoolpending = web3.manager.request_blocking('parity_pendingTransactions',[])
  File "/home/bang/.local/lib/python3.6/site-packages/web3/manager.py", line 106, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method parity_pendingTransactions does not exist/is not available'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bang/Documents/ethgasstation-backend/egs/main.py", line 29, in master_control
    txpool.append_current_txp()
  File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 152, in append_current_txp
    console.warn(e)
  File "/home/bang/Documents/ethgasstation-backend/egs/output.py", line 39, in warn
    msg = self._pad(msg)
  File "/home/bang/Documents/ethgasstation-backend/egs/output.py", line 70, in _pad
    lines = string.split("\n")
AttributeError: 'ValueError' object has no attribute 'split'

[2019-04-12 16:30:14] [INFO]  getting txpool hashes at block 0 ...
[2019-04-12 16:30:14] [INFO]  Switching pending tx acquisition method to geth
[2019-04-12 16:30:14] [ERROR] Traceback (most recent call last):
  File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 129, in _get_pending_tx_hashes
    txpoolpending = web3.manager.request_blocking('parity_pendingTransactions',[])
  File "/home/bang/.local/lib/python3.6/site-packages/web3/manager.py", line 106, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method parity_pendingTransactions does not exist/is not available'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 146, in append_current_txp
    hashlist = self._get_pending_tx_hashes()
  File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 138, in _get_pending_tx_hashes
    return self._get_pending_tx_hashes(False)
  File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 122, in _get_pending_tx_hashes
    txpoolcontent = web3.txpool.content
  File "/home/bang/.local/lib/python3.6/site-packages/web3/txpool.py", line 9, in content
    return self.web3.manager.request_blocking("txpool_content", [])
  File "/home/bang/.local/lib/python3.6/site-packages/web3/manager.py", line 106, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method txpool_content does not exist/is not available'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bang/Documents/ethgasstation-backend/egs/main.py", line 29, in master_control
    txpool.append_current_txp()
  File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 152, in append_current_txp
    console.warn(e)
  File "/home/bang/Documents/ethgasstation-backend/egs/output.py", line 39, in warn
    msg = self._pad(msg)
  File "/home/bang/Documents/ethgasstation-backend/egs/output.py", line 70, in _pad
    lines = string.split("\n")
AttributeError: 'ValueError' object has no attribute 'split'

[2019-04-12 16:30:14] [INFO]  getting txpool hashes at block 0 ...
[2019-04-12 16:30:14] [INFO]  Switching pending tx acquisition method to parity
[2019-04-12 16:30:14] [ERROR] Traceback (most recent call last):

The cause of the error

 File "/home/bang/Documents/ethgasstation-backend/egs/egs_ref.py", line 122, in _get_pending_tx_hashes
    txpoolcontent = web3.txpool.content
  File "/home/bang/.local/lib/python3.6/site-packages/web3/txpool.py", line 9, in content
    return self.web3.manager.request_blocking("txpool_content", [])
  File "/home/bang/.local/lib/python3.6/site-packages/web3/manager.py", line 106, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method txpool_content does not exist/is not available'}

I try to view txpool.content

$ geth  console
...other info...
> txpool.content
{
  pending: {},
  queued: {}
}

( Current GETH synchronization information is not complete )

I found two similar questions, But not solved.

#36

Has anyone encountered the same problem? Ask for help

Issue inserting into blockdata2

Hi,
I've been having some issues trying to run the backend. I'm new to Python and have been having problems debugging this one:

File "/home/jonathan/ethgasstation-backend/egs/main.py", line 240, in update_dataframes write_to_sql(alltx, block_sumdf, mined_blockdf, block) File "/home/jonathan/ethgasstation-backend/egs/main.py", line 76, in write_to_sql block_sumdf.to_sql(con=engine, name='blockdata2', if_exists='append', index=False) File "/home/jonathan/.local/lib/python3.5/site-packages/pandas/core/generic.py", line 1534, in to_sql chunksize=chunksize, dtype=dtype) File "/home/jonathan/.local/lib/python3.5/site-packages/pandas/io/sql.py", line 473, in to_sql chunksize=chunksize, dtype=dtype) File "/home/jonathan/.local/lib/python3.5/site-packages/pandas/io/sql.py", line 1156, in to_sql table.insert(chunksize) File "/home/jonathan/.local/lib/python3.5/site-packages/pandas/io/sql.py", line 670, in insert self._execute_insert(conn, keys, chunk_iter) File "/home/jonathan/.local/lib/python3.5/site-packages/pandas/io/sql.py", line 645, in _execute_insert conn.execute(self.insert_statement(), data) File "/home/jonathan/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 945, in execute return meth(self, multiparams, params) File "/home/jonathan/.local/lib/python3.5/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/home/jonathan/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement compiled_sql, distilled_params File "/home/jonathan/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context context) File "/home/jonathan/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception exc_info File "/home/jonathan/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/home/jonathan/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 186, in reraise raise value.with_traceback(tb) File "/home/jonathan/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context context) File "/home/jonathan/.local/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute cursor.execute(statement, parameters) File "/home/jonathan/.local/lib/python3.5/site-packages/mysql/connector/cursor.py", line 547, in execute stmt, self._process_params_dict(params)) File "/home/jonathan/.local/lib/python3.5/site-packages/mysql/connector/cursor.py", line 414, in _process_params_dict "Failed processing pyformat-parameters; %s" % err) sqlalchemy.exc.ProgrammingError: (mysql.connector.errors.ProgrammingError) Failed processing pyformat-parameters; Python 'hexbytes' cannot be converted to a MySQL type [SQL: 'INSERT INTO blockdata2 (miner, blockhash, includedblock, block_number, mingasprice, time_mined, gaslimit, blockfee, numtx, gasused, speed, main) VALUES (%(miner)s, %(blockhash)s, %(includedblock)s, %(block_number)s, %(mingasprice)s, %(time_mined)s, %(gaslimit)s, %(blockfee)s, %(numtx)s, %(gasused)s, %(speed)s, %(main)s)'] [parameters: {'miner': '0xa0a0f44f1b7a9dbf2e7a7b3622a497302d30d9944527f17c9ea5edd1d71d9400', 'blockhash': HexBytes('0x2b2c08a4cf048b2f0cd9ad463a6826c84ce20d611e8c6a53e4f53494aec5ab38'), 'numtx': None, 'block_number': 313269, 'mingasprice': None, 'time_mined': 0, 'gaslimit': 14995606, 'blockfee': 0.0, 'includedblock': None, 'main': 1, 'speed': 0.0, 'gasused': 0}]

any ideas why?

Thank you

show more more than 10 gas guzzlers

i'd like to be able to get data on more gas guzzlers than the top 10 at https://ethgasstation.info/gasguzzlers.php

anyway you could enable a GET parameter a la ?limit=x&offset=y for me to request this info remotely?

Model retraining / crossvalidation

Hello!

FIrst of all, thank you for sharing the code of such an important project -- it means a lot both for cryptoenthusiasts and open source community.

The file model_gas.py is addressing the minedtx2 table, the last reference to which was removed with 7ed90ad commit.

Problem with json_exporter

Is not producing valid output when value of parameter is nan

Add script to create database

It seems like this supposes that mysql has a database (tx) and some tables. Unless I missed something, there seems to be no way to create those automatically, or is there?

In case there's not, I'd suggest adding a script (or some migrations directory that can be used with some tool) that creates the database and the proper tables.

how long should I wait

when I ran

python3 ethgasstation.py
blocks 0
txcount 0

it just makes me think that it's doing something, but when I query tables it doesn't seem like it's doing anything. maybe I need to wait a bit more time or some progress update would be great to show in the output console

Dockerfile broken in latest build

Looks like model_gas.py was removed but the Dockerfile was not updated

 ---> 5e8b97a2a082
Step 2/10 : RUN apt-get update
 ---> Using cache
 ---> 7f317e35f344
Step 3/10 : RUN apt-get install -y python3 python3-pip
 ---> Using cache
 ---> 417ae6c24150
Step 4/10 : ADD settings.docker.conf /etc/ethgasstation.conf
 ---> Using cache
 ---> dff04e332e1d
Step 5/10 : ADD ethgasstation.py /opt/ethgasstation/ethgasstation.py
 ---> Using cache
 ---> dcdaa0e86633
Step 6/10 : ADD model_gas.py /opt/ethgasstation/model_gas.py
ADD failed: stat /var/lib/docker/tmp/docker-builder568375362/model_gas.py: no such file or directory
Exited with code 1```

website shows raw PHP

Hi EthGasStation!

This morning i noticed your website (http://ethgasstation.info/) only shows the PHP code, hope you can find the issue and fix it soon.
I notice it works on https:// maybe you can make a forced redirect from http to https

Thank you for making this awesome website!

CORS header for https://ethgasstation.info/json/ethgasAPI.json

Adding a CORS header that allows calls from other domains would make
https://ethgasstation.info/json/ethgasAPI.json
more useful.
Was there are reason it wasn't added, or was that just an oversight ?

Thanks!

Convert to pub/sub

New way:
To reduce effort on this, I'd like to see a feature branch of ethgasstation-backend that moves transactions away from the filter API to use pub/sub instead. This should be substantially less work than a muxer/load balancer. There is websocket support for web3.py. We should just use this.

Old bug:
Web3.py sucks. Hard. Filtering hammers Geth nodes that are already unreliable to begin with in the cloud.

We should build an intermediary layer for the Geth nodes that subscribes to txpools and blocks between multiple geth nodes, pushes all of the txpool data to redis pub/sub, all block information to MariaDB, and then the backend can be refactored with a class that simply reads from our existing data stores.

I have started an architecture for this in JS since the JS implementations for geth interface are useful, but if someone else wants to work on this in JS, Python or Golang I'll code review and help maintain any of them.

So I'm looking for a collaborator on this tool. I'm putting up the bounty for this one.

JSON-RPC Specs Issue

According to JSON-RPC spec version field name should be without a hyphen:

 {"jsonrpc": "2.0", "method": "subtract", "params": [42, 23], "id": 1}

In noonceBatch it is correct:

ethgasstation-backend/egs/txbatch.py

Line 71 in 7d662b0

'jsonrpc':'2.0',

While in batchRequest:

ethgasstation-backend/egs/txbatch.py

Line 38 in 7d662b0

'json-rpc': '2.0',

I am working on a porting the code to use with Parity and it was the issue as it seems that Parity is more strict regarding following the json-rpc specs

Ignore miner transactions

Miners sometimes include their own transactions in their mined blocks. Those transactions should be excluded so the hashpower acceptance would be more accurate. Normally those transactions have gas price of 1 Gwei.

After several "Bad Gateway" errors ETH Gas Station stops doing its job when Ethereum node is recovered

We're running Docker version of ETH Gas Station with our own Ethereum node (geth v1.8.22-stable). When something happens with Ethereum node (sometimes it happens when the node is restarted), Gas Station logs several errors (there can be over 30 errors at the same second). Errors look like this:

[2019-02-18 17:22:14] [ERROR] Traceback (most recent call last):
  File "/opt/ethgasstation/egs/main.py", line 29, in master_control
    txpool.append_current_txp()
  File "/opt/ethgasstation/egs/egs_ref.py", line 143, in append_current_txp
    current_block = web3.eth.blockNumber
  File "/usr/local/lib/python3.6/dist-packages/web3/eth.py", line 101, in blockNumber
    return self.web3.manager.request_blocking("eth_blockNumber", [])
  File "/usr/local/lib/python3.6/dist-packages/web3/manager.py", line 103, in request_blocking
    response = self._make_request(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/manager.py", line 86, in _make_request
    return request_func(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/gas_price_strategy.py", line 18, in middleware
    return make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 50, in apply_formatters
    response = make_request(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/attrdict.py", line 18, in middleware
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 50, in apply_formatters
    response = make_request(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/normalize_errors.py", line 9, in middleware
    result = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 50, in apply_formatters
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/formatting.py", line 50, in apply_formatters
    response = make_request(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/middleware/exception_retry_request.py", line 80, in middleware
    return make_request(method, params)
  File "/usr/local/lib/python3.6/dist-packages/web3/providers/rpc.py", line 67, in make_request
    **self.get_request_kwargs()
  File "/usr/local/lib/python3.6/dist-packages/web3/utils/request.py", line 22, in make_post_request
    response.raise_for_status()
  File "/usr/local/lib/python3.6/dist-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: <URL of our Ethereum node>

When Ethereum node is recovered, we see next logs in Docker container:

[2019-02-18 17:22:14] [INFO]  getting txpool hashes at block 7236900 ...
[2019-02-18 17:22:14] [INFO]  done. length = 0
[2019-02-18 17:22:14] [INFO]  listening for new transactions at block 7236900....
[2019-02-18 17:22:14] [WARN]  Pending transaction filter missing, re-establishing filter

And after that nothing else happens. Container with Gas Station is still alive, but it stops doing its job until it's manually restarted.

Consider tx size

Some transactions (for example https://etherscan.io/tx/0xec99eaff12b7100ac5428aa0737a12750362236da0ebd535b3f37077f8a743d0 ) consume large proportions of the block gas limit. These transactions will take longer to confirm than smaller transactions because they have to outbid larger portions of the pending tx pool. The current model does not take tx size into account. In particular, the Gas Used cell on the main page should affect confirmation time in addition to tx fee.

ImportError: No module named 'colored'

colored==1.3.5 package is missing in requirements.txt, and when you're trying to run this app as a Docker container, it says ImportError: No module named 'colored'.

NaNs

Yesterday (Jul 14) the network went crazy, the gas prices went up to 100 gwei and higher, and both latest branches ethgasstation-backend and oracle were predicting 'nan' for some values. Also, occasionally it would cause an exception. Here's some excerpt from the log:

{'fast': 1420, 'fastest': 1470, 'nomine': None, 'safeLow': nan, 'average': nan, 'block_time': 11.692307692307692, 'blockNum': 5963675,
 'speed': 0.99726616439117566}
[2018-07-14 18:03:59] [INFO] getting txpool hashes at block 5963680 ...
[2018-07-14 18:04:43] [INFO] done. length = 126590
[2018-07-14 18:04:43] [INFO] listening for new transactions at block 5963681....
[2018-07-14 18:04:45] [INFO] now processing block 5963676
[2018-07-14 18:04:45] [INFO] submitted tx: 0
[2018-07-14 18:04:46] [INFO] num mined in 5963673 = 80
[2018-07-14 18:04:46] [INFO] num seen in 5963673 = 8
[2018-07-14 18:04:46] [INFO] # of tx submitted ~ 5mago = 135
[2018-07-14 18:04:46] [INFO] # of tx submitted ~ 30mago = 406
[2018-07-14 18:04:46] [INFO] txpool block length 17977
[2018-07-14 18:04:46] [INFO] safelow: 1000
[2018-07-14 18:04:46] [INFO] avg: 1030
cannot convert float NaN to integer
[2018-07-14 18:04:46] [ERROR]Traceback (most recent call last):
  File "/root/ethgasstation-backend/egs/main.py", line 59, in master_control
    gaspricereport.get_wait(predictiontable.predictiondf)
  File "/root/ethgasstation-backend/egs/egs_ref.py", line 735, in get_wait
    self.gprecs['safeLowWait'] = lookup(self.gprecs['safeLow'])
  File "/root/ethgasstation-backend/egs/egs_ref.py", line 733, in lookup
    wait = round(wait, 1)
UnboundLocalError: local variable 'wait' referenced before assignment

Error in _get_pending_tx_hashes

I haven't been able to get this backend to run properly. I'm sure its something small that I forgot to do or simply overlooked, so any help/pointers would be greatly appreciated!

Running on Ubuntu 16.04 with python 3.5.2 and geth 1.8.12.
Starting my geth instance with geth --verbosity 3 --testnet --rpc --rpcapi "eth,net,personal,web3" --syncmode "light" console 2>> /tmp/eth.log

And running the ethgasstation with python3 ethgasstation.py

The console output is the following (repeating):

[2018-08-07 15:57:34] [INFO]  getting txpool hashes at block 3792490 ...
[2018-08-07 15:57:34] [INFO]  Switching pending tx acquisition method to parity
[2018-08-07 15:57:34] [ERROR] Traceback (most recent call last):
  File "/home/sam/ethgasstation-backend/egs/egs_ref.py", line 119, in _get_pending_tx_hashes
    txpoolcontent = web3.txpool.content
  File "/home/sam/.local/lib/python3.5/site-packages/web3/txpool.py", line 9, in content
    return self.web3.manager.request_blocking("txpool_content", [])
  File "/home/sam/.local/lib/python3.5/site-packages/web3/manager.py", line 110, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method txpool_content does not exist/is not available'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sam/ethgasstation-backend/egs/egs_ref.py", line 143, in append_current_txp
    hashlist = self._get_pending_tx_hashes()
  File "/home/sam/ethgasstation-backend/egs/egs_ref.py", line 135, in _get_pending_tx_hashes
    return self._get_pending_tx_hashes(False)
  File "/home/sam/ethgasstation-backend/egs/egs_ref.py", line 126, in _get_pending_tx_hashes
    txpoolpending = web3.manager.request_blocking('parity_pendingTransactions',[])
  File "/home/sam/.local/lib/python3.5/site-packages/web3/manager.py", line 110, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method parity_pendingTransactions does not exist/is not available'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sam/ethgasstation-backend/egs/main.py", line 29, in master_control
    txpool.append_current_txp()
  File "/home/sam/ethgasstation-backend/egs/egs_ref.py", line 149, in append_current_txp
    console.warn(e)
  File "/home/sam/ethgasstation-backend/egs/output.py", line 39, in warn
    msg = self._pad(msg)
  File "/home/sam/ethgasstation-backend/egs/output.py", line 70, in _pad
    lines = string.split("\n")
AttributeError: 'ValueError' object has no attribute 'split'

[2018-08-07 15:57:34] [INFO]  getting txpool hashes at block 3792490 ...
[2018-08-07 15:57:34] [INFO]  Switching pending tx acquisition method to geth
[2018-08-07 15:57:34] [ERROR] Traceback (most recent call last):
  File "/home/sam/ethgasstation-backend/egs/egs_ref.py", line 126, in _get_pending_tx_hashes
    txpoolpending = web3.manager.request_blocking('parity_pendingTransactions',[])
  File "/home/sam/.local/lib/python3.5/site-packages/web3/manager.py", line 110, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method parity_pendingTransactions does not exist/is not available'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sam/ethgasstation-backend/egs/egs_ref.py", line 143, in append_current_txp
    hashlist = self._get_pending_tx_hashes()
  File "/home/sam/ethgasstation-backend/egs/egs_ref.py", line 135, in _get_pending_tx_hashes
    return self._get_pending_tx_hashes(False)
  File "/home/sam/ethgasstation-backend/egs/egs_ref.py", line 119, in _get_pending_tx_hashes
    txpoolcontent = web3.txpool.content
  File "/home/sam/.local/lib/python3.5/site-packages/web3/txpool.py", line 9, in content
    return self.web3.manager.request_blocking("txpool_content", [])
  File "/home/sam/.local/lib/python3.5/site-packages/web3/manager.py", line 110, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method txpool_content does not exist/is not available'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sam/ethgasstation-backend/egs/main.py", line 29, in master_control
    txpool.append_current_txp()
  File "/home/sam/ethgasstation-backend/egs/egs_ref.py", line 149, in append_current_txp
    console.warn(e)
  File "/home/sam/ethgasstation-backend/egs/output.py", line 39, in warn
    msg = self._pad(msg)
  File "/home/sam/ethgasstation-backend/egs/output.py", line 70, in _pad
    lines = string.split("\n")
AttributeError: 'ValueError' object has no attribute 'split'

EDIT: It was an easy fix... Just forgot to put txpool in geth's --rpcapi flag

write_to_json: Exception caught: 10 is not JSON serializable

Hello,

I am running an instance of ethgasstation-backend, pointed at redis/mariadb/fully sync'd geth node with txpool enabled. Here are my settings:

[mysql]
    hostname  = localhost
    port      = 3306
    username  = ethgasstation
    password  = <SNIP>
    database  = mysql

[redis]
    hostname  = localhost
    port      = 6379
    protocol  = redis

[geth]
    hostname = <SNIP>
    port     = 80
    protocol = http

[json]
    output_type = redis
    output_location = http://localhost:6379

[api]
    behind_reverse_proxy = false
    proxy_real_ip_header = x-forwarded-for
    rate_limit_disable = false
    rate_limit_max_requests = 120
    rate_limit_request_window_seconds = 60
    banhammer_disable = false
    banhammer_backing_store = redis
    banhammer_api_key_file = /tmp/api_ban.txt
    banhammer_ip_addr_file = /tmp/ip_ban.txt

I am noticing that there are no keys populating in the Redis instance. In addition, I am receiving more or less the same error constantly as each block is processed:

[2018-07-11 14:39:22] [INFO]  getting txpool hashes at block 5945239 ...
[2018-07-11 14:39:23] [INFO]  done. length = 4096
[2018-07-11 14:39:23] [INFO]  listening for new transactions at block 5945239....
[2018-07-11 14:39:35] [INFO]  now processing block 5945239
[2018-07-11 14:39:35] [INFO]  submitted tx: 1052
[2018-07-11 14:39:36] [INFO]  added tx: 76
[2018-07-11 14:39:36] [INFO]  added tx: 75
[2018-07-11 14:39:37] [INFO]  num mined in 5945236 = 189
[2018-07-11 14:39:37] [INFO]  num seen in 5945236 = 102
[2018-07-11 14:39:37] [INFO]  # of tx submitted ~ 5mago = 969
[2018-07-11 14:39:37] [INFO]  # of tx submitted ~ 30mago = 0
[2018-07-11 14:39:37] [INFO]  txpool block length 692
[2018-07-11 14:39:38] [INFO]  safelow: 90
[2018-07-11 14:39:38] [INFO]  avg: 100
[2018-07-11 14:39:38] [INFO]  updating 151 transactions
[2018-07-11 14:39:38] [ERROR] write_to_json: Exception caught: 120 is not JSON serializable

Any ideas? I will try reverting an older version of the codebase and report back.

JSON Serialization Issues Not Fixed

Duplicate of #26
The issue is not fixed as of the latest release nor the latest commit on master.

[2018-07-13 14:09:24] [INFO]  submitted tx: 178
[2018-07-13 14:09:25] [INFO]  added tx: 56
[2018-07-13 14:09:25] [INFO]  num mined in 5957144 = 111
[2018-07-13 14:09:25] [INFO]  num seen in 5957144 = 0
[2018-07-13 14:09:25] [INFO]  # of tx submitted ~ 5mago = 0
[2018-07-13 14:09:25] [INFO]  # of tx submitted ~ 30mago = 0
[2018-07-13 14:09:25] [INFO]  txpool block length 85
[2018-07-13 14:09:25] [INFO]  safelow: 10
[2018-07-13 14:09:25] [INFO]  avg: 110
[2018-07-13 14:09:25] [INFO]  updating 56 transactions
{'safeLow': 10.0, 'avgWait': 8.3, 'speed': 0.97069827481120519, 'average': 60.0, 'fastest': 110, 'safeLowWait': 8.3, 'blockNum': 5957147, 'fastestWait': 0.5, 'nomine': None, 'fastWait': 0.5, 'block_time': 14.0, 'fast': 110}
[2018-07-13 14:09:25] [ERROR] write_to_json: Exception caught: 110 is not JSON serializable```

A test suite is needed

This project is starting to actually have traction, so it's time for us to start writing some tests for functionality. This is a placeholder issue to discuss what should have tests, what framework we should use, et cetera.

Updated json/ethgasAPI.json throws 403

Seems an update to json/ethgasAPI.json was deployed and it now is returning HTTP 403 Forbidden response when accessed by a service.

Is something more now needed to fetch this .json file? If so, is this documented somewhere?

Thanks!

Smart contract holding current price

Hey, by any chance, do you have (or plan to have) a deployed smart contract that holds the last gas price and is updated by this oracle? Thanks.

Parity compatibility

Unfortunately the Parity compatibility is currently blocked by the lack of web3.txpool support in the Parity RPC:

Traceback (most recent call last):
  File "ethgasstation.py", line 354, in update_dataframes
    current_txpool = get_txhases_from_txpool(block)
  File "ethgasstation.py", line 95, in get_txhases_from_txpool
    txpoolcontent = web3.txpool.content
  File "/Users/me/.virtualenvs/ethgasstation-adaptive-oracle/lib/python3.6/site-packages/web3/txpool.py", line 9, in content
    return self.web3.manager.request_blocking("txpool_content", [])
  File "/Users/me/.virtualenvs/ethgasstation-adaptive-oracle/lib/python3.6/site-packages/web3/manager.py", line 96, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'Method not found'}

API Documentation / units?

https://ethgasstation.info/json/ethgasAPI.json

Hey guys, I'm trying to use your adaptive oracle API, but can't figure out what Units they're in. It seems like 10x Gwei, but not 100%, can anyone confirm that these are the units? 70gwei for safe seems super high and right now it's 7 on the site.