firepol / ccxt-websockets-db-updater Goto Github PK

A Python 3 command line program to subscribe cryptocurrency exchanges websockets and save order books in real time in a database.

License: MIT License

Python 100.00%

ccxt-websockets-db-updater's Introduction

CCXT websockets DB updater

A Python 3 command line program to subscribe cryptocurrency exchanges websockets and save order books in real time in a database.

This project is based on CCXT lfern/websockets-multiple fork.

To speed up installation, as the CCXT repository is huge and takes a long time to download, I created a repository containing just the transpiled files from lfern's branch: https://github.com/firepol/ccxt-websockets When the branch will be forked in the official CCXT project, this temporary branch won't be needed anymore.

Note: in the requirements.txt file, in the 1st line you see a dependency to the ccxt-websockets repository, which could not be up to date. In case you want to update it yourself, you have to

fork the ccxt-websockets repository
clone lfern's fork
checkout lfern's websockets-multiple branch
build lfern's websockets-multiple branch as follows (if you prefer npm than yarn, adjust the commands accordingly):
```
yarn
yarn export-exchanges
yarn transpile
```
overwrite the python folder of the ccxt-websockets repository with the one you just updated; do the same with the package.json file
commit and push (you must add the remote pointing to your own fork and push there)
change the 1st line of the requirements.txt to point to your fork of the ccxt-websockets repository
pip uninstall ccxt
pip install -r requirements.txt

Credits for the websockets implementations: lfern and his collaborators.

If you are looking for an alternative in Java, check my other similar project: Cryptows.

Installation

# Clone this repository
git clone https://github.com/firepol/ccxt-websockets-db-updater.git

# Enter the repository dir
cd ccxt-websockets-db-updater

# Create a `data` directory (needed to store exchange settings and websockets you want to subscribe to)
mkdir data

# Copy the sample settings.ini into data/
cp samples/settings.ini data/

# Create a python 3.6 virtual environment
virtualenv -p /usr/bin/python3.6 env

# Activate the python virtual environment
source ~/env/ccxtws/bin/activate

# Install all required dependencies
pip install -r requirements.txt

Update

To get the latest updates from https://github.com/firepol/ccxt-websockets:

# Unintall the old version of CCXT
pip uninstall ccxt

# Get all missing dependencies (this will fetch the latest CCXT version linked in requirements.txt)
pip install -r requirements.txt

Quick start

Note: If you are using Windows the python commands written in this document may not work for you. In that case just call the command prefixed with python, like this:

python script_name.py

Inside the repo, you can run this command to get an overview of all the command line parameters:

./ob_tester.py -h

Then test the BTC/USDT websocket connection to binance:

./ob_tester.py -e binance -s ETH/BTC --verbose

Configuration

First of all, copy samples/settings.ini to data/settings.ini.

[config]: this is the main configuration section.

db_url: database connection string.

Examples:

PostgreSQL: postgres://postgres:postgres@localhost:5432/cryptows
SQLite: sqlite:///cryptows.db

order_book_entries: (by default: 1) limit of order books to fetch (e.g. 5 means 5 bids and 5 asks).

Exchanges and pairs you want to fetch:in square brackets write the exchange name, in the following line insert a list of symbols, as follows:

[bitstamp]
symbols: ETH/BTC
         BTC/USD
         ETH/USD

You can comment (with ";" at the beginning of the line) sections and single lines, e.g.:

[bitstamp]
symbols: ETH/BTC
;         BTC/USD
         ETH/USD

Some exchanges (like cex.io) need to be authenticated via api key / secret. Add your api keys in the data/exchange_api_keys.json file (you find a template in the samples folder).

Usage

The usage of the websocket database updater is quite straightforward:

./ob_updater.py

The program checks the exchanges configured in the data/settings.ini file and creates a websocket connection for each pair.

Add --debug to see when an order book update occurs.
Add --verbose to see the order book update values. Add --reset_db to delete all records in the order_book table before subscribing the websockets.

License

MIT License

ccxt-websockets-db-updater's People

Contributors

Stargazers

Watchers

Forkers

fialtbit rockrockyy samulic minusworld tobby2002 bw-xu ulandz fengfei01 bazickoff dangleburry bellyfat trial123zel

ccxt-websockets-db-updater's Issues

db schema created but no data inserted

Hi firepol,
thanks for this wrapper!!!

I am trying to demo the db persist part and it connects fine, creates the schema but does not insert data

NVM its the tester script I was running - quite impressive I must admit!

python3 ./ob_tester.py -e cex -s ETH/BTC = websocket error

(myenv) ➜ ccxt-websockets-db-updater git:(master) ✗ python3 ./ob_tester.py -e cex -s ETH/BTC --verbose

Could not find ./data/exchange_api_keys.json
------------------------------>
wss://ws.cex.io/ws/
ParseResult(scheme='wss', netloc='ws.cex.io', path='/ws/', params='', query='', fragment='')
cex, 2019-03-09 13:39:10.371241, [<FrameSummary file ./ob_tester.py, line 45 in >, <FrameSummary file ./ob_tester.py, line 34 in main>, <FrameSummary file /usr/lib/python3.6/asyncio/base_events.py, line 427 in run_forever>, <FrameSummary file /usr/lib/python3.6/asyncio/base_events.py, line 1440 in _run_once>, <FrameSummary file /usr/lib/python3.6/asyncio/events.py, line 145 in _run>, <FrameSummary file /home/oem/python/ccxt-websockets-db-updater/ob_updater.py, line 103 in websocket_error>]

Fix multiple subscriptions for binance

If I configure multiple pairs for binance, it doesn't work.

To reproduce, configure:
settings_ws.json:

{
  "limit": 5,
  "subscriptions": {
    "binance": [
      "NANO/BTC", "ETH/BTC"
    ]
  }
}

Then run:

./ob_updater.py --debug

Error:

Task exception was never retrieved
future: <Task finished coro=<subscribe_ws() done, defined at /home/firepol/projects-python/ccxtws/utils.py:40> exception=Exception('connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)',)>
Traceback (most recent call last):
  File "/home/firepol/projects-python/ccxtws/utils.py", line 58, in subscribe_ws
    await exchange.websocket_subscribe(event, symbol, {'limit': limit})
  File "/home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/base/exchange.py", line 786, in websocket_subscribe
    conxid = await self._websocket_ensure_conx_active(event, symbol, True, params)
  File "/home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/base/exchange.py", line 621, in _websocket_ensure_conx_active
    await self.websocket_connect(conxid)
  File "/home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/base/exchange.py", line 648, in websocket_connect
    await websocket_connection.connect()
  File "/home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/websocket_connection.py", line 94, in connect
    await future
Exception: connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)

Expected result, like in the other exchanges, the program should work, not crash for multiple pairs in the same exchange.

AttributeError: 'poloniex' object has no attribute 'on'

Good morning, I've the following error with any exchange like POLONIEX or CEX for example.

[2020-09-17 11:31:52,712, ERROR] Task exception was never retrieved
future: <Task finished name='Task-1' coro=<subscribe_ws() done, defined at C:\Users\33651\Desktop\websocket poloniex\ob_updater.py:89> exception=AttributeError("'poloniex' object has no attribute 'on'")>
Traceback (most recent call last):
File "C:\Users\33651\Desktop\websocket poloniex\ob_updater.py", line 102, in subscribe_ws
@exchange.on('err')
AttributeError: 'poloniex' object has no attribute 'on'

When execute ./ob_tester.py --debug the command then get this error

Traceback (most recent call last):
File "./ob_tester.py", line 8, in
import ob_updater
File "/home/davidsun/ccxt-websockets-db-updater/ob_updater.py", line 13, in
import book_utils
File "/home/davidsun/ccxt-websockets-db-updater/book_utils.py", line 6, in
import utils
File "/home/davidsun/ccxt-websockets-db-updater/utils.py", line 5, in
import ccxt.async_support as ccxt
File "/home/davidsun/ccxt-websockets-db-updater/env/lib/python3.6/site-packages/ccxt/init.py", line 183, in
from ccxt.poloniex import poloniex # noqa: F401
File "/home/davidsun/ccxt-websockets-db-updater/env/lib/python3.6/site-packages/ccxt/poloniex.py", line 1273
else:
^
IndentationError: expected an indented block

Handle Timeout errors

To reproduce, run ./ob_updater.py with a settings.ini configured with a lot of pairs. Here the stacktrace.

ob bitfinex2 XRP/BTC, None: ask [9.038e-05, -500]; bid: [9.033e-05, 226.91324961]
ob bitfinex2 XRP/BTC, None: ask [9.038e-05, -500]; bid: [9.033e-05, 226.91324961]
ob bitfinex2 XRP/BTC, None: ask [9.038e-05, -500]; bid: [9.033e-05, 226.91324961]
Task exception was never retrieved
future: <Task finished coro=<subscribe_ws() done, defined at /home/firepol/projects-python/ccxtws/utils.py:48> exception=TimeoutError('timeout in scope: websocket_subscribe',)>
Traceback (most recent call last):
File "/home/firepol/projects-python/ccxtws/utils.py", line 84, in subscribe_ws
await exchange.websocket_subscribe(event, symbol, {'limit': limit})
File "/home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/base/exchange.py", line 805, in websocket_subscribe
await future
TimeoutError: timeout in scope: websocket_subscribe
Task exception was never retrieved
future: <Task finished coro=<subscribe_ws() done, defined at /home/firepol/projects-python/ccxtws/utils.py:48> exception=TimeoutError('timeout in scope: websocket_subscribe',)>
Traceback (most recent call last):
File "/home/firepol/projects-python/ccxtws/utils.py", line 84, in subscribe_ws
await exchange.websocket_subscribe(event, symbol, {'limit': limit})
File "/home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/base/exchange.py", line 805, in websocket_subscribe
await future
TimeoutError: timeout in scope: websocket_subscribe
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
/usr/lib/python3.6/asyncio/base_events.py:498: RuntimeWarning: coroutine 'MyClientProtocol.onMessage' was never awaited
self._ready.clear()
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Task was destroyed but it is pending!
task: <Task pending coro=<MyClientProtocol.onMessage() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/websocket/pusher_light_connection.py:61>>
Closing Loop
After complete
coinbasepro requires to release all resources with an explicit call to the .close() coroutine. If you are creating the exchange instance from within your async coroutine, add exchange.close() to your code into a place when you're done with the exchange and don't need the exchange instance anymore (at the end of your async coroutine).
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fbdb6522fd0>
Fatal write error on socket transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fbda6ea6b70>
transport: <_SelectorSocketTransport fd=15>
Traceback (most recent call last):
File "/usr/lib/python3.6/asyncio/selector_events.py", line 763, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fbda6ea6b70>
transport: <_SelectorSocketTransport closing fd=15>
Traceback (most recent call last):
File "/usr/lib/python3.6/asyncio/selector_events.py", line 763, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/asyncio/sslproto.py", line 650, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.6/asyncio/selector_events.py", line 767, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.6/asyncio/selector_events.py", line 645, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.6/asyncio/selector_events.py", line 657, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.6/asyncio/base_events.py", line 580, in call_soon
self._check_closed()
File "/usr/lib/python3.6/asyncio/base_events.py", line 366, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
bitfinex2 requires to release all resources with an explicit call to the .close() coroutine. If you are creating the exchange instance from within your async coroutine, add exchange.close() to your code into a place when you're done with the exchange and don't need the exchange instance anymore (at the end of your async coroutine).
bitstamp requires to release all resources with an explicit call to the .close() coroutine. If you are creating the exchange instance from within your async coroutine, add exchange.close() to your code into a place when you're done with the exchange and don't need the exchange instance anymore (at the end of your async coroutine).
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fbdb6d93668>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fbdb6510ac8>
Task was destroyed but it is pending!
task: <Task pending coro=<subscribe_ws() running at /home/firepol/projects-python/ccxtws/utils.py:84> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fbdb6e0b768>()]>>
Task was destroyed but it is pending!
task: <Task pending coro=<throttle..run() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/base/throttle.py:41> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fbda6f056a8>()]>>
Task was destroyed but it is pending!
task: <Task pending coro=<Exchange._executeAndCallback..t() running at /home/firepol/env/ccxtws/lib/python3.6/site-packages/ccxt/async_support/base/exchange.py:747> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fbda6f05678>()]>>
binance requires to release all resources with an explicit call to the .close() coroutine. If you are creating the exchange instance from within your async coroutine, add exchange.close() to your code into a place when you're done with the exchange and don't need the exchange instance anymore (at the end of your async coroutine).
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fbdb6d93a90>
Future exception was never retrieved
future:
TimeoutError

Fix ob_updater subscribe_ws error handling

In ob_updater, subscribe_ws I removed these lines and added a TODO to this issue.

    error_message = type(err).__name__ + ":" + str(err)
    logging.error(f'{exchange.id}: {error_message}')
    logging.error(error_stack)

The problem is that the error is not logged. Try to run binance and cexio at the same time to get some errors.

Also, I removed these lines:

    await exchange.close()
    raise WsError(exchange.id)

This was causing binance websockets to stop. My idea was to close the channel with the problematic exchange/pair, not shut down ALL channels from one exchange.

Ideally I'd like to re-subscribe the websocket that is causing errors (or that got disconnected).

To reproduce:

settings.ini

[config]
db_url: postgres://dbusr:dbpwd@localhost:5432/cryptows
order_book_entries_limit: 1

[binance]
symbols: BTC/USDT
         EOS/USDT
         ETH/USDT

[cex]
symbols: BTC/USD
         BTC/EUR

Run:

./ob_updater.py --reset_db

Error I get:

subscribed: cex BTC/USD
subscribed: cex BTC/EUR
subscribed: binance BTC/USDT
binance, 2019-01-17 23:27:57.255584, [<FrameSummary file /home/paul/projects-python/ccxtws/ob_updater.py, line 142 in >, <FrameSummary file /home/paul/projects-python/ccxtws/ob_updater.py, line 68 in main>, <FrameSummary file /usr/lib/python3.6/asyncio/base_events.py, line 427 in run_forever>, <FrameSummary file /usr/lib/python3.6/asyncio/base_events.py, line 1440 in _run_once>, <FrameSummary file /usr/lib/python3.6/asyncio/events.py, line 145 in _run>, <FrameSummary file /home/paul/projects-python/ccxtws/ob_updater.py, line 88 in websocket_error>]
subscribed: binance EOS/USDT
subscribed: binance ETH/USDT

Fix database issues

Websockets order books are flooding the database. The problem is that SQLAlchemy doens't seem thread safe as I read, or not supporting async.

E.g. bitfinex "takes over" and order books from other exchanges are not saved anymore. But I see no error messages in the logs. So I think the WS connection works, it's just the part of savign to the DB that seems to be skipped for those.

Try a different postgres driver or wrapper.

E.g.:

Then get rid of the hack (that doesn't even work properly) introduced in this commit: 7e2dca1

Implications: queries may be rewritten, but the goal is still to keep using SQLAlchemy ORM, to allow using different databases.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.