Comments (5)
BTW @TalhaAsmal I'm doing my recent work on the #memory branch, which tracks TensorForce's #memory branch. In case you're diving in deep, might benefit from their recent updates
from tforce_btc_trader.
@TalhaAsmal I did a heavy-handed force-push to #memory. I was in flux using that as a personal dev branch (I'm now using #tmp as my dirty branch). I won't be force-pushing to #memory anymore, so you can work from it safely - sorry for that.
You'll want to rebase your work to it (easiest is probably: (1) create a new branch from #memory, (2) cherry-pick your own branch's commits one-by-one). After following some conversation on TensorForce's Gitter channel, looks like 0.3.5.1 is pretty much non-functional (you won't get any models to converge); that the commits in tensorforce#memory fix many problems. That's to say, our tforce_btc_trader#master is non-functional.
A big noteable recent change. tensorforce#memory switches Policy Gradient models' batching from timestep-batching to episode-batching. Timestep-batching gave us control over explicit batch sizes (1024 means 1024 timesteps, aka 1024 rows/datapoints to send to TensorFlow). Episodes on the other hand are however many timesteps the agent stays alive (doesn't dip below 0 value/cash, doesn't HODL too long). Could be 10k timesteps; could be 100k. Turned out this buffer would fill up pretty fast and crash my GPU (1080 ti, aint nothing to shake a stick at). In order to make episode-batching work, I had to reduce a lot of dimensional parameters. Reduced batch_size
, max steps per episode (from 20k to 6k; also corresponding punish_repeats
), and importantly the dimensionality of each timestep's state (aka the features). I added an AutoEncoder to reduce a timestep's dimensions from ~22 (plus/minus a bunch depending on arbitrage & indicators) to 6. Pretty damn destructive. I've got some ideas to improve this situation going forward:
- Improve the autoencoder's performance. Currently has a ~2% MSE, if we can widdle then maybe it's find to do what we're doing.
- Switch from PPO to DDPG, which I believe uses timestep-batching instead of episode. It's a new model in tensorforce#memory, and actually one I've seen used for trading. An important bit is to stick with continuous actions, so it can buy/sell whatever amount. eg, DQNs require discrete actions.
- Move to a different framework (see #11)
from tforce_btc_trader.
I've been testing out the memory branch of this and tensorforce since last night, so not much to stay yet.
I've seen the following warning message a couple of times on the first run of hypersearch.py, but not after:
2018-02-10 02:51:38.030610: W C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 516.70MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
It's also taking REALLY long on my gtx 1070. Only 3 runs in about 12 hours, but I'll leave it running probably until Monday at least to see how it performs.
Also, I'm running with a custom data set (pulled from Bitfinex API, only has OHCLV data) with 133986 rows of data (Just about 3 months data). Is this enough, or should I pump more data into it?
from tforce_btc_trader.
@TalhaAsmal do a pull, added some stuff (changed some DB structure, mind). Also pull the latest lefnire/tensorforce#memory (or reinforceio/tensorforce#memory, they're 1-1 currently). Note, you won't be able to do gpu-split right now; they've temporarily disabled a single runner's session_config
. That actually might fix your memory issue! I have a commit manually splitting in 2 for my 1080 ti, but you'd likely wanna avoid that given the issue.
The issue being RAM-maxing. Both the warning, and the amount of time it takes. Indeed, the way tensorforce#memory handles batches w/ PG models is as episode-batches rather than step-batches, which massively increases the load vs #master (hence my big rant above). So I'm not surprised you're facing that issue. This commit was a heavy-handed attempt to mitigate that; I'm surprised it didn't cut if for your case. This commit reverts the auto-encoder, so if you're still playing w/ the trader & facing memory issues, uncomment the lines here.
Since my comment, I tried DDPG in hopes to switch to timestep-batching (and save RAM). The agent doesn't currently support multi-state configs (dict-based states), so will need to revisit in the future. Will continue posting w/ updates
from tforce_btc_trader.
I noticed very strange GPU usage behaviour, shown below:
My usage will be ~8%-10% for a long while, then suddenly spike and just as suddenly decrease. I've never seen this before in my NN training, and I wonder if that's why performance is so bad. Is there any way to profile or debug this behaviour?
I'm running another hypersearch run tonight to see if the memory error persists.
from tforce_btc_trader.
Related Issues (20)
- Error: Cannot find module 'webpack' HOT 3
- ModuleNotFoundError: No module named 'data.data' HOT 2
- (psycopg2.ProgrammingError) relation "coinbase" does not exist HOT 3
- Live-mode NotImplementedError HOT 3
- Switching to conv1d/ LSTM-FCN representation
- Table runs does not exist HOT 5
- Visualisation issue HOT 3
- Non-existent input ^ConstantFoldingCtrl/ppo/cond/add/Switch_0 for node ConstantFolding/ppo/cond/strided_slice_2/stack_const_axis HOT 11
- Data Source HOT 1
- Volatile GPU-Util is low HOT 6
- Can we add a support and resistance indicator? HOT 1
- Actions exploration HOT 4
- v0.2 Save trials object to resume on next run HOT 2
- Python hung (likely because of too many psycopg2 connections) during attempted impmentation of https://github.com/deepmind/scalable_agent HOT 1
- Suggestion: setup docker to be easy to develop. HOT 2
- Can't find run.py? HOT 1
- Google Colab Tutorial not showing
- Typeerror occurred at tensorforce 0.4.3 When run hypersearch.py
- How to add new RL features from media news to train them?
- This is a question HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tforce_btc_trader.