lefnire / tforce_btc_trader Goto Github PK

Current implementation is very crude - just enough to ballpark strategy & hyper combo. Backtesting is custom-built. It relies on next-state close prices for PNL; buys/sells on close (instead of buying at ASK and selling at BID); only supports market-orders as such (no stop/limits); etc. Then there's more complex bits like order depth (how much can I buy at this price before that guy runs out and I have to fill the rest of that order at a different price). Very importantly, w/o stop/limit orders, there's no risk-control, which is half the game (could double the agent's performance). (Accounting for order types will require augmenting the Env's action-space).

All this could be solved w/ a backtesting library, and it's high-time. I've scoped out a few, my notes here. I'm leaning on backtrader currently, and possibly paired with ccxt

Going on faith that hypersearch.py / run.py are even acting the way they're supposed to. Really truly need unit tests in this project. Ain't no weekend project, this is money we're dealing with.

It shows error while trying "python run.py --name test".
What could be the cause of this (issue/problem)?
Appreciate your help.

Traceback (most recent call last):
File "run.py", line 57, in
main()
File "run.py", line 43, in main
**hydrated
File "c:\users\administrator\rl\tensorforce\tensorforce\agents\ppo_agent.py", line 151, in init
entropy_regularization=entropy_regularization
File "c:\users\administrator\rl\tensorforce\tensorforce\agents\learning_agent.py", line 149, in init
batching_capacity=batching_capacity
File "c:\users\administrator\rl\tensorforce\tensorforce\agents\agent.py", line 79, in init
self.model = self.initialize_model()
File "c:\users\administrator\rl\tensorforce\tensorforce\agents\ppo_agent.py", line 179, in initialize_model
likelihood_ratio_clipping=self.likelihood_ratio_clipping
File "c:\users\administrator\rl\tensorforce\tensorforce\models\pg_prob_ratio_model.py", line 88, in init
gae_lambda=gae_lambda
File "c:\users\administrator\rl\tensorforce\tensorforce\models\pg_model.py", line 95, in init
requires_deterministic=False
File "c:\users\administrator\rl\tensorforce\tensorforce\models\distribution_model.py", line 86, in init
discount=discount
File "c:\users\administrator\rl\tensorforce\tensorforce\models\memory_model.py", line 106, in init
reward_preprocessing=reward_preprocessing
File "c:\users\administrator\rl\tensorforce\tensorforce\models\model.py", line 200, in init
self.setup()
File "c:\users\administrator\rl\tensorforce\tensorforce\models\model.py", line 307, in setup
self.initialize(custom_getter=custom_getter)
File "c:\users\administrator\rl\tensorforce\tensorforce\models\pg_model.py", line 107, in initialize
super(PGModel, self).initialize(custom_getter)
File "c:\users\administrator\rl\tensorforce\tensorforce\models\distribution_model.py", line 93, in initialize
kwargs=dict(summary_labels=self.summary_labels)
File "c:\users\administrator\rl\tensorforce\tensorforce\core\networks\network.py", line 180, in from_spec
kwargs=kwargs
File "c:\users\administrator\rl\tensorforce\tensorforce\util.py", line 159, in get_object
return obj(*args, **kwargs)
File "C:\Users\Administrator\RL\tforce_btc_trader\hypersearch.py", line 155, in init
super(CustomNet, self).init(layers_spec, **kwargs)
File "c:\users\administrator\rl\tensorforce\tensorforce\core\networks\network.py", line 270, in init
kwargs=dict(scope=scope, summary_labels=summary_labels)
File "c:\users\administrator\rl\tensorforce\tensorforce\core\networks\layer.py", line 143, in from_spec
kwargs=kwargs
File "c:\users\administrator\rl\tensorforce\tensorforce\util.py", line 159, in get_object
return obj(*args, **kwargs)
TypeError: init() got an unexpected keyword argument 'cell_clip'

It's going to be a few days before I can test this, but if anyone wants give this a shot and report back it would be greatly appreciated. As long as there isn't anything heavily nested being passed back in trials, this should work. Tested with hyperas but should work with hyperopt. Saving the pickle to a local file is a temporary solution until it can be pushed to a SQL table and pulled back down.

Somewhere at the top of hypersearch add:
import pickle

Then replace lines 344-346 with this:

    # set initial max_eval, attempt to load a saved trials object from pickle, if that fails start fresh.
    # grab how many trials were previously run and add max_evals to it for the next run.
    # this allows the hyper parameter search to resume where it left off last.
    # TODO save trials to SQL table and restore from there instead of local pickle. 
    max_evals = 20
    try:
        trialPickle = open('./trial.pickle','rb')
        trials = pickle.load(trialPickle)
        max_evals = len(trials.trials) + max_evals
    except:
        trials = Trials()

    best = fmin(loss_fn, space=space, algo=tpe.suggest, max_evals=max_evals, trials=trials)

    with open('./trial.pickle', 'wb') as f:
            pickle.dump(trials, f)

Hyperopt seems to support saving this data to mongoDB, however, we can probably get it to a json friendly format and keep the data in an sql table similar to the runs
https://github.com/hyperopt/hyperopt/wiki/Parallelizing-Evaluations-During-Search-via-MongoDB

I'm running node 8.11.1 on ubuntu 16.04. Flask server is running and returning on port 500 but when I run the npm start I get a config issue.

npm start => localhost:8080
✖ ｢wds｣: Invalid configuration object. Webpack has been initialised using a configuration object that does not match the API schema.

• configuration.entry should be one of these:
  object { : non-empty string | [non-empty string] } | non-empty string | [non-empty string] | function
  -> The entry point(s) of the compilation.

Traceback (most recent call last):
File "kaggle.py", line 7, in
from data.data import engine
ModuleNotFoundError: No module named 'data'

Could some body help? Thanks.

Hi, where can i find this dependency?
from box import Box

Hi,

I get the following error and stacktrace when running hypersearch.py. I'm using Python 3.6.4 64-bit on Windows 10 with Postgres 10.1.3 64-bit. I've installed all the requirements in the requirements.txt file with the same version as specified, but I still get the following error.

2018-02-02 22:59:09.443734: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX
2018-02-02 22:59:09.708364: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.7845
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.64GiB
2018-02-02 22:59:09.708511: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
row_count: 1574274
Traceback (most recent call last):
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\engine\base.py", line 1112, in _execute_context
conn = self.__connection
AttributeError: 'Connection' object has no attribute '_Connection__connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\engine\base.py", line 1114, in _execute_context
conn = self._revalidate_connection()
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\engine\base.py", line 429, in _revalidate_connection
raise exc.ResourceClosedError("This Connection is closed")
sqlalchemy.exc.ResourceClosedError: This Connection is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "hypersearch.py", line 775, in
main()
File "hypersearch.py", line 771, in main
y_list=Y
File "I:\toolkits\tforce_btc_trader_clean\gp.py", line 193, in bayesian_optimisation2
y_list.append(loss_fn(params))
File "hypersearch.py", line 722, in loss_fn
reward = hsearch.execute(vec2hypers(params))
File "hypersearch.py", line 549, in execute
env.train_and_test(agent, self.cli_args.n_steps, self.cli_args.n_tests, -1)
File "I:\toolkits\tforce_btc_trader_clean\btc_env.py", line 466, in train_and_test
self.use_dataset(Mode.TEST)
File "I:\toolkits\tforce_btc_trader_clean\btc_env.py", line 273, in use_dataset
df = data.db_to_dataframe(self.conn, limit=limit, offset=offset, arbitrage=self.hypers.arbitrage)
File "I:\toolkits\tforce_btc_trader_clean\data\data.py", line 201, in _db_to_dataframe_main
df = pd.read_sql_query(query, conn).iloc[::-1]
File "i:\python_venv\tforce_btc_trader\lib\site-packages\pandas\io\sql.py", line 332, in read_sql_query
parse_dates=parse_dates, chunksize=chunksize)
File "i:\python_venv\tforce_btc_trader\lib\site-packages\pandas\io\sql.py", line 1087, in read_query
result = self.execute(*args)
File "i:\python_venv\tforce_btc_trader\lib\site-packages\pandas\io\sql.py", line 978, in execute
return self.connectable.execute(*args, **kwargs)
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\engine\base.py", line 939, in execute
return self._execute_text(object, multiparams, params)
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\engine\base.py", line 1097, in _execute_text
statement, parameters
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\engine\base.py", line 1121, in _execute_context
None, None)
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\engine\base.py", line 1402, in _handle_dbapi_exception
exc_info
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\util\compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\util\compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\engine\base.py", line 1114, in _execute_context
conn = self._revalidate_connection()
File "i:\python_venv\tforce_btc_trader\lib\site-packages\sqlalchemy\engine\base.py", line 429, in _revalidate_connection
raise exc.ResourceClosedError("This Connection is closed")
sqlalchemy.exc.StatementError: (sqlalchemy.exc.ResourceClosedError) This Connection is closed [SQL: 'select coinbase.open as coinbase_open, coinbase.high as coinbase_high, coinbase.low as coinbase_low, coinbase.close as coinbase_close, coinbase.volume_btc as coinbase_volume_btc, coinbase.volume_currency as coinbase_volume_currency, coinbase.weighted_price as coinbase_weighted_price, coincheck.open as coincheck_open, coincheck.high as coincheck_high, coincheck.low as coincheck_low, coincheck.close as coincheck_close, coincheck.volume_btc as coincheck_volume_btc, coincheck.volume_currency as coincheck_volume_currency, coincheck.weighted_price as coincheck_weighted_price from coinbase\n left join lateral (\n select open, high, low, close, volume_btc, volume_currency, weighted_price\n from coincheck\n where coincheck.timestamp <= coinbase.timestamp\n order by coincheck.timestamp desc\n limit 1 \n ) coincheck on true\n order by coinbase.timestamp desc limit 157427 offset 1416846']

Is this a postgres version issue, or something related to windows? It seems to happen when doing the Select statement for the test data.

I am testing with a GTX1070 . when running python hypersearch.py to hypersearch ,By running nvidia-smi , I found that the Volatile GPU-Util is very low (only 6%) with Perf = P2 and GPU memory used 7773MB.

The hyper search speed is also not faster than cpu very much , I expected the speed should be 10x. Is it correct?

best regards

I am trying to get started with the project and I am getting the following error:

python run.py --name test

Traceback (most recent call last):
  File "run.py", line 57, in <module>
    main()
  File "run.py", line 50, in main
    env.train_and_test(agent, args.n_steps, args.n_tests, args.early_stop)
  File "/Users/bvz/Documents/tforce_btc_trader/btc_env.py", line 599, in train_and_test
    runner.run(timesteps=timesteps_each)
  File "/Users/bvz/Documents/tensorforce/tensorforce/execution/runner.py", line 126, in run
    state, terminal, reward = self.environment.execute(actions=action)
  File "/Users/bvz/Documents/tforce_btc_trader/btc_env.py", line 462, in execute
    custom = self.end_episode_score()
  File "/Users/bvz/Documents/tforce_btc_trader/btc_env.py", line 545, in end_episode_score
    sharpe = self.sharpe()
  File "/Users/bvz/Documents/tforce_btc_trader/btc_env.py", line 532, in sharpe
    diff = (pd.Series(totals.trade).pct_change() - pd.Series(totals.hold).pct_change())[1:]
  File "/Users/bvz/anaconda3/lib/python3.6/site-packages/pandas/core/series.py", line 227, in __init__
    raise_cast_failure=True)
  File "/Users/bvz/anaconda3/lib/python3.6/site-packages/pandas/core/series.py", line 2868, in _sanitize_array
    subarr = _possibly_convert_platform(data)
  File "/Users/bvz/anaconda3/lib/python3.6/site-packages/pandas/core/common.py", line 1002, in _possibly_convert_platform
    values = lib.list_to_object_array(values)
TypeError: Argument 'obj' has incorrect type (expected list, got BoxList)

I am working out of the memory branch.

I have installed all of the dependencies, gathered the data from kaggle, imported it into postgres, and updated the config.json.

Is this a data or dependency issue on my end?

Running hypersearch.py is giving me this error:
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "runs" does not exist
LINE 1: select hypers, returns from runs where flag='conv2d'
^
[SQL: 'select hypers, returns from runs where flag=%(f)s'] [parameters: {'f': 'conv2d'}]

Which looks to me like the runs table does not exist in hyper_runs - which makes sense as there seems to be nothing trying to create it?

I've created the table but I don't know what data types to use.

Update: Bigint[] everything apart from agent and flag which are text and it works!

Update 2: I'm not seeing any of the runs being written to the table and looking at the dataset I'm seeing on the console I'm assuming I have the datatypes very wrong - any view of what the datatypes are would be great.

FYI using Ubuntu 16.04, python 3.6

The graph assembles just fine, after finalization and onto training, one of the threads (assuming main?) gets hung. This is the traceback:

-------------------- Thread 4590249408 --------------------
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/training/queue_runner_impl.py", line 257, in _run
enqueue_callable()
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1257, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)

also get this traceback for the same thread (getting hung here as well?):

File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/training/queue_runner_impl.py", line 293, in _close_on_stop
coord.wait_for_stop()
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 311, in wait_for_stop
return self._stop_event.wait(timeout)
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/threading.py", line 551, in wait
signaled = self._cond.wait(timeout)
File "/Users/hughalessi/miniconda3/envs/rl/lib/python3.6/threading.py", line 295, in wait
waiter.acquire()

Thread traceback acquired using https://gist.github.com/niccokunzmann/6038331. I'm assuming this is because each instance of the btc_env creates a new psycopg2 connection to the history database hosted by postgresql, however I do not know how to fix it.

Although I know this is an outside project, any insight would be greatly appreciated. If the deepMind IMPALA implementation demonstrates decent results, I would be happy to share the implementation here if I can get it working.

Hi,

The following error occurs right at the end of a training run when running hypersearch.py with no other arguments:

100% Custom: 0.885 Sharpe: 0.000 Return: 0.000 Trades: 0[<0] 10000[=0] 0[>0]
Running no-kill test-set
Traceback (most recent call last):
File "hypersearch.py", line 852, in
main()
File "hypersearch.py", line 848, in main
y_list=Y
File "I:\toolkits\tforce_btc_trader\gp.py", line 193, in bayesian_optimisation2
y_list.append(loss_fn(params))
File "hypersearch.py", line 799, in loss_fn
reward = hsearch.execute(vec2hypers(params))
File "hypersearch.py", line 624, in execute
env.train_and_test(agent, self.cli_args.n_steps, self.cli_args.n_tests, -1)
File "I:\toolkits\tforce_btc_trader\btc_env.py", line 619, in train_and_test
self.run_deterministic(runner, print_results=True)
File "I:\toolkits\tforce_btc_trader\btc_env.py", line 587, in run_deterministic
next_state, terminal, reward = self.execute(runner.agent.act(next_state, deterministic=True))
File "I:\toolkits\tforce_btc_trader\btc_env.py", line 425, in execute
pct_change = self.prices_diff[step_acc.i + 1]
IndexError: index 12864 is out of bounds for axis 0 with size 12864

TensorForce's memory branch supposedly fixes a major flaw in the PPO implementation. It's a pretty wild-west branch for now, and I've started my own branch to follow it. I'm gonna put that on hold until they merge their branch into master and cut a release. We'll need to adapt to the new hyperparameters introduced (adding them to our hypersearch framework).

I'm getting the following error by running python3 run.py --id 10 --name test --test

  File "tforce_btc_trader/env/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "coinbase" does not exist
LINE 1: ...se.weighted_price as coinbase_weighted_price from coinbase o...
                                                             ^
 [SQL: 'select coinbase.open as coinbase_open, coinbase.high as coinbase_high, coinbase.low as coinbase_low, coinbase.close as coinbase_close, coinbase.volume_btc as coinbase_volume_btc, coinbase.volume_currency as coinbase_volume_currency, coinbase.weighted_price as coinbase_weighted_price from coinbase order by timestamp desc limit 100000 offset 0'] (Background on this error at: http://sqlalche.me/e/f405)

Is it profitable or test application? How is real world trading? I just download. I will try to run my spare time.

It shows error while trying "5. Visualize".
What could be the cause of this (issue/problem)?
Appreciate your help.
("npm install;npm install -g webpack-dev-server"comand has NO ERROR)

npm start => localhost:8080
module.js:478
throw err;
^

Error: Cannot find module 'webpack'
at Function.Module._resolveFilename (module.js:476:15)
at Function.Module._load (module.js:424:25)
at Module.require (module.js:504:17)
at require (internal/module.js:20:19)
at Object. (/usr/lib/node_modules/webpack-dev-server/lib/Server.js:22:17)
at Module._compile (module.js:577:32)
at Object.Module._extensions..js (module.js:586:10)
at Module.load (module.js:494:32)
at tryModuleLoad (module.js:453:12)
at Function.Module._load (module.js:445:3)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: webpack-dev-server "="
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

log
0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'start', '=' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info lifecycle [email protected]prestart: [email protected]
6 info lifecycle [email protected]start: [email protected]
7 verbose lifecycle [email protected]start: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected]start: PATH: /usr/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/ubuntu/tforce_btc_trader/visualize/client/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
9 verbose lifecycle [email protected]start: CWD: /home/ubuntu/tforce_btc_trader/visualize/client
10 silly lifecycle [email protected]start: Args: [ '-c', 'webpack-dev-server "="' ]
11 silly lifecycle [email protected]start: Returned: code: 1 signal: null
12 info lifecycle [email protected]start: Failed to exec start script
13 verbose stack Error: [email protected] start: webpack-dev-server "="
13 verbose stack Exit status 1
13 verbose stack at EventEmitter. (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:285:16)
13 verbose stack at emitTwo (events.js:106:13)
13 verbose stack at EventEmitter.emit (events.js:191:7)
13 verbose stack at ChildProcess. (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack at emitTwo (events.js:106:13)
13 verbose stack at ChildProcess.emit (events.js:191:7)
13 verbose stack at maybeClose (internal/child_process.js:920:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:230:5)
14 verbose pkgid [email protected]
15 verbose cwd /home/ubuntu/tforce_btc_trader/visualize/client
16 verbose Linux 4.4.0-1052-aws
17 verbose argv "/usr/bin/node" "/usr/bin/npm" "start" "="
18 verbose node v6.13.1
19 verbose npm v5.7.1
20 error code ELIFECYCLE
21 error errno 1
22 error [email protected] start: webpack-dev-server "="
22 error Exit status 1
23 error Failed at the [email protected] start script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

Just got here and read the readme. Data sourcing is no problem with BTC or any other crypto. We can get it straight from the exchanges for free. Check out the library CCXT.

Per #6 (comment), I'd like to try the DDPG RL agent (compared to PPO agent). DDPG hypers will need to be added to hypersearch, and likely some other code adjustments. I once had DQN support, when I removed it I may have tailored the code to be too PPO-centric.

Assuming i want to add web scraper from top news sources to train new features to make bot aware not only candlestick but financial news that infer market situation.
How do i do that?

1. Perhaps switch to conv1d instead of using conv2d layers in hyperparameter.py ? I'm pretty sure conv1d is just a layer on conv2d, but it's probably more reliable if things change in the future since i believe Tensorforce calls tensorflow's conv layers.

Have you tried using this representation:
https://arxiv.org/abs/1801.04503
for your data?

I'm working on a similar problem equities and this representation was giving me somewhat better results on the data. It gives the network two views of the data (detailed in the paper), which seems to help.

If you put the project into a docker container it will be easier to run and develop and even to get more help

[Update 2018-07-27] Update: seems Coach has slowed down (w/o much community), and rllab has stopped. A more recently popular framework is rllib (one letter different than rllab).

I'd like to try replacing TensorForce with Coach and see if we get any better performance. Use the Clipped PPO for an apples-to-apples comparison; instructions on converting our btc_env.py here.

History: before landing on TensorForce I'd tried rll/rllab and openai/baselines. Baselines is backed by OpenAI, the company behind half these algorithms (they're behind PPO, the model we're using). But baselines isn't a plug-n-play framework intended for developer use; instead it's a dumping-ground for each paper's corresponding sample code. I couldn't get any of their stuff customized to our use-case; all runs resulted eventually in NaNs everywhere. Coach is new. No hunch as to whether it'll outperform, though it is backed by Intel which bodes well. I want to give it a whirl, but don't have time right now - so if anyone wants to take a stab at it, please indeed!

It shows error while trying "python run.py --name test".
(This is other repository's command "lefnire/tforce_btc_trader.git")
What could be the cause of this (issue/problem)?
Appreciate your help.

Traceback (most recent call last):
File "run.py", line 57, in
main()
File "run.py", line 43, in main
**hydrated
File "c:\users\administrator\rl\tensorforce\tensorforce\agents\ppo_agent.py", line 151, in init
entropy_regularization=entropy_regularization
File "c:\users\administrator\rl\tensorforce\tensorforce\agents\learning_agent.py", line 149, in init
batching_capacity=batching_capacity
File "c:\users\administrator\rl\tensorforce\tensorforce\agents\agent.py", line 79, in init
self.model = self.initialize_model()
File "c:\users\administrator\rl\tensorforce\tensorforce\agents\ppo_agent.py", line 179, in initialize_model
likelihood_ratio_clipping=self.likelihood_ratio_clipping
File "c:\users\administrator\rl\tensorforce\tensorforce\models\pg_prob_ratio_model.py", line 88, in init
gae_lambda=gae_lambda
File "c:\users\administrator\rl\tensorforce\tensorforce\models\pg_model.py", line 95, in init
requires_deterministic=False
File "c:\users\administrator\rl\tensorforce\tensorforce\models\distribution_model.py", line 86, in init
discount=discount
File "c:\users\administrator\rl\tensorforce\tensorforce\models\memory_model.py", line 106, in init
reward_preprocessing=reward_preprocessing
File "c:\users\administrator\rl\tensorforce\tensorforce\models\model.py", line 200, in init
self.setup()
File "c:\users\administrator\rl\tensorforce\tensorforce\models\model.py", line 307, in setup
self.initialize(custom_getter=custom_getter)
File "c:\users\administrator\rl\tensorforce\tensorforce\models\pg_model.py", line 107, in initialize
super(PGModel, self).initialize(custom_getter)
File "c:\users\administrator\rl\tensorforce\tensorforce\models\distribution_model.py", line 93, in initialize
kwargs=dict(summary_labels=self.summary_labels)
File "c:\users\administrator\rl\tensorforce\tensorforce\core\networks\network.py", line 180, in from_spec
kwargs=kwargs
File "c:\users\administrator\rl\tensorforce\tensorforce\util.py", line 159, in get_object
return obj(*args, **kwargs)
File "C:\Users\Administrator\RL\tforce_btc_trader\hypersearch.py", line 155, in init
super(CustomNet, self).init(layers_spec, **kwargs)
File "c:\users\administrator\rl\tensorforce\tensorforce\core\networks\network.py", line 270, in init
kwargs=dict(scope=scope, summary_labels=summary_labels)
File "c:\users\administrator\rl\tensorforce\tensorforce\core\networks\layer.py", line 143, in from_spec
kwargs=kwargs
File "c:\users\administrator\rl\tensorforce\tensorforce\util.py", line 159, in get_object
return obj(*args, **kwargs)
TypeError: init() got an unexpected keyword argument 'cell_clip'

For some reason when I import your new tutorial into Google Colab, only the headings show and none of the code. I can't figure it out.

Currently the conv2d set uses time as the x-axis (naturally), and features as the y-axis (not-so-naturally). Then we have window/stride sort of finding patterns positionally. Makes perfect sense to have a window finding positional patterns on the time axis, but really the features should all boil down together, not in chunks. So TODO: experiment with features as channels/depth instead of height.

I have the following issue by running the koggle.py script

Traceback (most recent call last):
  File "data/populate/kaggle.py", line 10, in <module>
    from data.data import engine
ModuleNotFoundError: No module named 'data.data'

I just run the "python hypersearch.py" directly and get the following error. Actually, I find that "self.monitored_session.run" have repeated many times and finally it got a None. I am not familiar with tensorforce and do not know what happened. Any help is appreciated.

Traceback (most recent call last):
  File "hypersearch.py", line 772, in <module>
    main()
  File "hypersearch.py", line 768, in main
    y_list=Y
  File "/home/RL/tforce_btc_trader/gp.py", line 193, in bayesian_optimisation2
    y_list.append(loss_fn(params))
  File "hypersearch.py", line 719, in loss_fn
    reward = hsearch.execute(vec2hypers(params))
  File "hypersearch.py", line 547, in execute
    env.train_and_test(agent, self.cli_args.n_steps, self.cli_args.n_tests, -1)
  File "/home/RL/tforce_btc_trader/btc_env.py", line 487, in train_and_test
    self.run_deterministic(runner, print_results=True)
  File "/home/RL/tforce_btc_trader/btc_env.py", line 471, in run_deterministic
    runner.agent.act(next_state, deterministic=False)
  File "/home/anaconda3/lib/python3.6/site-packages/tensorforce/agents/agent.py", line 145, in act
    deterministic=deterministic
  File "/home/anaconda3/lib/python3.6/site-packages/tensorforce/models/model.py", line 1268, in act

    actions, internals, timestep = self.monitored_session.run(fetches=fetches, feed_dict=feed_dict)
  File "/home/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 536, in run
    return self._sess.run(fetches,
AttributeError: 'NoneType' object has no attribute 'run'

This is a TODO. so I create it as an issue to fix if I figure it out

File "run.py", line 48, in main env.run_live(agent, test=args.test_live) File "tforce_btc_trader/btc_env.py", 
line 438, in run_live self.run_deterministic(runner, print_results=True) File 
"tforce_btc_trader/btc_env.py", line 389, in run_deterministic next_state, terminal, reward = 
self.execute(runner.agent.act(next_state, deterministic=True, independent=True)) File 
"tforce_btc_trader/btc_env.py", line 343, in execute raise NotImplementedError NotImplementedError

** hint

See 6fc4ed2 for prior live-mode code which worked. Much has changed since then and it won't work inthat state, so removing and leaving to you to fix (and submit PR please!)

Hey,

I'm trying to set the project up and am running into a problem when populating the data.

When running python data/populate/kaggle.py as described as part of step 2 I'm seeing this error message.

$ python data/populate/kaggle.py

Traceback (most recent call last):
  File "data/populate/kaggle.py", line 7, in <module>
    from data.data import engine
ModuleNotFoundError: No module named 'data'

My first guess was that it had to have something to do with the working directory. So i tried some different ones, but still can't get it to work.

Any of you guys having the same issue?

Cheers

File "kaggle.py", line 30
df = pd.read_csv(f'./bitcoin-historical-data/{filename}.csv')
^
SyntaxError: invalid syntax

Could somebody give me a hand? Thanks.

I just run hypersearch.py with no argument then get the following error.
I am not familiar with tensorforce but I know it happened statement in tensorforce/execution/runner.py which show how episodes progress.

I could run hypersearch.py with my fixed tensorforce which is commentted out this error state btw.

My environment

• Tensorforce 0.4.3
• Tensorflow-gpu 1.13.1
• tforce_btc_trader v0.2

the error

Traceback (most recent call last):
  File "hypersearch.py",
 line 364, in <module>
    main()
  File "hypersearch.py", line 357, in main
    best = fmin(loss_fn, space=space, algo=tpe.suggest, max_evals=max_evals, trials=trials)
  File "/usr/local/lib/python3.6/site-packages/hyperopt/fmin.py", line 388, in fmin
    show_progressbar=show_progressbar,
  File "/usr/local/lib/python3.6/site-packages/hyperopt/base.py", line 639, in fmin
    show_progressbar=show_progressbar)
  File "/usr/local/lib/python3.6/site-packages/hyperopt/fmin.py", line 407, in fmin
    rval.exhaust()
  File "/usr/local/lib/python3.6/site-packages/hyperopt/fmin.py", line 262, in exhaust
    self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
  File "/usr/local/lib/python3.6/site-packages/hyperopt/fmin.py", line 227, in run
    self.serial_evaluate()
  File "/usr/local/lib/python3.6/site-packages/hyperopt/fmin.py", line 141, in serial_evaluate
    result = self.domain.evaluate(spec, ctrl)
  File "/usr/local/lib/python3.6/site-packages/hyperopt/base.py", line 844, in evaluate
    rval = self.fn(pyll_rval)
  File "hypersearch.py", line 315, in loss_fn
    env.train_and_test(agent)
  File "/root/tforce_btc_trader/btc_env.py", line 255, in train_and_test
    runner.run(timesteps=train_steps)
  File "/root/tensorforce2/tensorforce/execution/runner.py", line 149, in run
    pbar.update(num_episodes - self.global_episode)
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

Create a PR to TensorForce adding this commit, which manually removes auto-closing agent/model w/in runner.py so that we can work with the model a bit after training (to test, to use in live-mode, etc) - we'll close manually.

Above commit is insufficient. Will need to change all their code instances currently using runner.py and adding agent.close();model.close() after runner.run(). This is the only commit in my fork required for this project, so getting that into upstream will remove my fork as a dep.

I'm working outside of hypersearch right now so these are probably not ideal parameters. It seems the model becomes a little more flexible to less than perfect parameters (and the random associated with the model's initial state) with actions exploration defined.

https://reinforce.io/blog/introduction-to-tensorforce/
actions_exploration=dict(
type='ornstein_uhlenbeck',
sigma=0.1,
mu=0.0,
theta=0.1
),
these parameters are from the example in the above link and are not optimized

Any benefit to adding parameters for actions exploration to hypersearch?