timothyyu / gdax-orderbook-ml Goto Github PK

save results of autosr() x4 (every 15min) to csv or file in conjunction with scrape time start period for data scrape

https://keras.io/layers/convolutional/

to look into:
keras flatten layer, permute layer, reshape layer, repeat vector

[pure size and price, potentially position also]

take l2 states as grid, feed into convolutional 2d, then flatten/reshape, then feed into lstm/gru as input in series

lambada functionality for specialized weighting of values near s/r lines?
https://github.com/keras-team/keras/blob/master/keras/layers/core.py#L564

https://pandas.pydata.org/pandas-docs/stable/io.html

https://www.h5py.org/

https://github.com/timothyyu/wsae-lstm

Roll improved aspects of project structure, methodology, and model design/data structures from WSAE-LSTM repository into this project after WSAE-LSTM model tested (eventually)

function may not be updating 'size' correctly between orderbook states; look into + fix

current requirements.txt is invalid/missing requirements

model structure + parameter output potential for #18 (dashboard)

pipreqs package doesn't work on windows; create new venv and requirements.txt

needs model image and screenshot(s) from ppt in default readme

volume as feature input from seperate model?
https://statcompute.wordpress.com/2017/01/08/an-example-of-merge-layer-in-keras/

https://drivendata.github.io/cookiecutter-data-science/

replace existing ".gdax.com" endpoint/websocket references with gdax prime/pro endpoints
https://docs.gdax.com/#coinbase-prime

https://github.com/transcranial/keras-js

this plus mlive/heroku + or a combination of dedicated or virtual VPS instance due to model bandwith and data size requirements (current)

rewrite autoSR() function to be more accurate to S/R done by hand

• matplotlib finance is deprecated

• use d3/plotly/seaborn or pure matplotlib;

• d3/plotly = actual usable modern charts

• related to #9

reconstruct/alter #6 jupyter notebook scrape file for 10 hours of scrape

issues:

• Ram limitations
• Mongo db raw scrape size
• csv limitations as a file format

input: OHLC data
output: autogenerated S/R levels

have be accessible via private authentication, limit requests when run on own hardware server

route using flask/heroku

missing folder with design sketches/notes and mockups

• boolean flag for scrape_running() status

• save to raw_data_pipeline folder

• timezone basis/reference config (timezone of scrape)

• outline of function def for new hour of data (mock function)

• outline of error handling if scrape interrupted

add animations and transitions for revision of ppt

1-5m sampling frames of data, prior to transfer learning
related to #47

1. scrape new set of 1hr test data with OHLC candlestick data saved to msgpack/csv additionally

2. related to "ch15m_req_time() not respecting time format" issue + autosr() results save:
   #9 & #33

after scrape, then parse into snapshots of orderbook at different states

closely related to #48 and iterator/sampling structure

• plotly, d3, flask, html
• bootstrap

n=200-500 from execution point to reduce memory/cpu/io overhead

see documentation, design_mockup folder:

Existing sample historical BTCUSD tick data from http://api.bitcoincharts.com/v1/csv/ failure to load + commit, even exceeding Git LFS file limit size.
File will not load under excel - use Text Editor/IDE:


vscode:

mongo db collection size limit reset in code (upon new size limit or collection start)

restrict range of query before groupby if necessary; potential insights into model behavior and orderbook s/r movement states with query/group combination

I want to offer a new point of view, and my colaboraty

Why this stock prediction project ?

Things this project offers that I did not find in other free projects, are:

• Testing with +-30 models. Multiple combinations features and multiple selections of models (TensorFlow , XGBoost and Sklearn )
• Threshold and quality models evaluation
• Use 1k technical indicators
• Method of best features selection (technical indicators)
• Categorical target (do buy, do sell and do nothing) simple and dynamic, instead of continuous target variable
• Powerful open-market-real-time evaluation system
• Versatile integration with: Twitter, Telegram and Mail
• Train Machine Learning model with Fresh today stock data

https://github.com/Leci37/stocks-prediction-Machine-learning-RealTime-telegram/tree/develop

https://github.com/csko/gdax-python-api

Research:
https://github.com/manahl/arctic

mongodb compatible; feather is not

https://git-lfs.github.com/

timestamps from first row of each l2update_15m are not being passed correctly; gdax-python api requires specific format for historical candlestick requests

limit n size for input batch for stateful = True
i.e. loop abstraction to split 10 hours of data into 1 hour each, or 15min x4 per hour for training and validation; related to train_test_split issue

Implementation/hooks to research:

https://travis-ci.org/
https://www.codacy.com
https://codecov.io/
https://coveralls.io/

Live mongo db instance for db instance, linked to herkou (potential):
https://mlab.com/ + https://www.heroku.com/

https://stackoverflow.com/questions/42861460/how-to-interpret-weights-in-a-lstm-layer-in-keras?utm_medium

keras-team/keras#3088

for e in zip(model.layers[0].trainable_weights, model.layers[0].get_weights()):
print('Param %s:\n%s' % (e[0],e[1]))

to look into - split data before reshape, manually split into train/test/split, or use another library that works with 3-dimensional input data

context vector for historical s/r awareness of price movement, either using single layer dense or single layer LSTM

implement either static (predefined, manually calc) or dynamic (live, on the fly calc)