hugocen / freqtrade-gym Goto Github PK

A customized gym environment for developing and comparing reinforcement learning algorithms in crypto trading.

License: GNU General Public License v3.0

Python 100.00%

freqtrade baseline algorithmic-trading reinforcement-learning gym machine-learning cryptocurrency crypto trading trading-bot

freqtrade-gym's Introduction

freqtrade-gym

This project is base on freqtrade

The project is in very early stage, so there are a lot of inconvenient part that you have to set up manually. I am working on the improvements.

Installation

1. freqtrade

Follow the freqtrade documentation to install freqtrade

Initialize the user_directory

freqtrade create-userdir --userdir user_data/

2. Pandas

pip install pandas

3. OpenAI Gym

pip install gym

4. Copy freqtrade-gym files

Baseline files

IndicatorforRL.py -> [freqtrade home]/user_data/strategies/IndicatorforRL.py
config_rl.json -> [freqtrade home]/config_rl.json
freqtradegym.py -> [freqtrade home]/freqtradegym.py
deep_rl.py -> [freqtrade home]/deep_rl.py

RLib

Copy first the baseline files.
LoadRLModel.py -> [freqtrade home]/user_data/strategies/LoadRLModel.py
rllib_example.py -> [freqtrade home]/rllib_example.py

Example Usage (baseline)

The usage example is deep_rl.py and the config for freqtrade and freqtrade-gym is config_rl.json and uses IndicatorforRL.py as feature extraction.
This demo is using openai baseline library to train reinforcement learning agents.
Baseline can install by

sudo apt-get update && sudo apt-get install cmake libopenmpi-dev python3-dev zlib1g-dev
pip install stable-baselines[mpi]

Download historical data
(Remember to download a little bit more data than the timerange in config file just in case.)

freqtrade download-data -c <config file> --days <Int> -t {1m,3m,5m...}

To match the example config_rl.json

freqtrade download-data -c config_rl.json --timerange 20201119-20201201 -t 15m

Move the IndicatorforRL.py into user_data/strategies (you should have user_data/strategies/IndicatorforRL.py)

Run the demo to train an agent.

python deep_rl.py

You can use tensorboard to monior the training process
logdir is defined in deep_rl.py when initializing the rl model

tensorboard --logdir <logdir>

This will look like

Example Usage (RLlib)

The usage example is rllib_example.py and the config for freqtrade and freqtrade-gym is config_rl.json and uses IndicatorforRL.py as feature extraction.
This demo is using RLlib to train reinforcement learning agents.
Baseline can install by

pip install 'ray[rllib]'

Run the demo to train an agent.

python rllib_example.py

Example of Loading model for backtesting or trading (baseline)

Move the LoadRLModel.py into user_data/strategies (you should have user_data/strategies/LoadRLModel.py)

Modified the class intial load model part to your model type and path.

Modified the populate_indicators and rl_model_redict method for your gym settings.

Run the backtesting

freqtrade backtesting -c config_rl.json -s LoadRLModel

Dry-run trading (remove --dry-run for real deal!)

freqtrade trade --dry-run -c config_rl.json -s LoadRLModelgProto

TODO

Update the strategy for loadinf the trained model for backtesting and real trading. (baseline)
The features name and total feature number(freqtradegym.py line 89) have to manually match in the indicator strategy and in freqtradegym. I would like to come up with a way to set up features in config file.
RLlib example.
Update the strategy for loadinf the trained model for backtesting and real trading (RLlib).
NEAT example.

DISCLAIMER

This software is for educational purposes only. Do not risk money which you are afraid to lose. USE THE SOFTWARE AT YOUR OWN RISK. THE AUTHORS AND ALL AFFILIATES ASSUME NO RESPONSIBILITY FOR YOUR TRADING RESULTS.

freqtrade-gym's People

Contributors

Stargazers

Watchers

Forkers

tutecode sergioisidoro mormar7 weaver-viii selskas fredrik81 cosmicmonkey888 otreewen2020 skytraders-fork ciaranwelsh t3ch9 raedkit snussik bjwie marcelsavegnago q-learning-trader virtueistheonlygood fdoperezi dankez dolphincodes juwarny ketzah lovehedy24 arby1108 alirezamdv prmts23 damrongsak pnmice ken2190 daboooooo meesokim trongtran-98 renhongquan ahmedsakrr exabyt3s 7da benwaldner cylphysics son1128 thanhen pixelioo quanterstudio veria70 systv gold3nfish viotemp1

freqtrade-gym's Issues

'IndicatorforRL' object has no attribute 'ohlcvdata_to_dataframe'

I have installed the files in freqtrade and get this error in freqtradegym.py at line 45, when I try to run deep_rl.py

Exception has occurred: AttributeError
'IndicatorforRL' object has no attribute 'ohlcvdata_to_dataframe'

I don't know from where comes this ohlcvdata_to_dataframe function, I cannot find it anywhere declared...

Can you advise on how to load the checkpoints from RLLib

RLLib seems to store the checkpoints under ~/ray_results ... but how do I load them for backtesting, this seems to be a different format?

Profitable training

Hi hugo, is it possible to have profitable modele after learning ? i can t have one

ModuleNotFoundError: No module named 'tensorflow.contrib'

What an interesting project, thanks for this. I've tried to run the demo to train an agent, but got this error:

~/freqtrade$ python deep_rl.py
2020-12-04 03:35:21.973385: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "deep_rl.py", line 8, in <module>
    from stable_baselines.common.policies import MlpPolicy
  File "/home/virtual/freqtrade/.env/lib/python3.8/site-packages/stable_baselines/__init__.py", line 19, in <module>
    from stable_baselines.ddpg import DDPG
  File "/home/virtual/freqtrade/.env/lib/python3.8/site-packages/stable_baselines/ddpg/__init__.py", line 2, in <module>
    from stable_baselines.ddpg.ddpg import DDPG
  File "/home/virtual/freqtrade/.env/lib/python3.8/site-packages/stable_baselines/ddpg/ddpg.py", line 11, in <module>
    import tensorflow.contrib as tc
ModuleNotFoundError: No module named 'tensorflow.contrib'

Is it related to this?

`tensorflow.contrib is being removed in version 2.0``

If so, I could not fix it with this:
https://github.com/deetungsten/stable-baselines

Any other ideas? Should I just revert to Python 3.7 and Tensorflow <2.x ?

python3 -c 'import tensorflow as tf; print(tf.__version__)'
I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2.3.1

don't work whit tensorflow 2

How to Use

Would you please share some instruction about how to use your code.
Thank You

How to use RLlib model for trading?

I have trained RLlib model and don't know how to use that model for live trading. Could you please help me with that?

Problem while executing python deep_rl.py command

(.env) root@freqtrade2:/home/screw/freqtrade# python deep_rl.py
Traceback (most recent call last):
File "deep_rl.py", line 7, in
from freqtradegym import TradingEnv
File "/home/screw/freqtrade/freqtradegym.py", line 16, in
from freqtrade.configuration import (TimeRange, remove_credentials,
ImportError: cannot import name 'remove_credentials' from 'freqtrade.configuration' (/home/screw/freqtrade/freqtrade/configuration/init.py)

The best backtests results

Hello, buds

I am test this on Tensorflow version and my results of backtests is not good enough for dry run mode tests. You can see it attached file: ft_rl_results-default.txt

Whats your results?

some questions

I have tried your example and it works!
Now i'd like to ask some questions for clarification

in freqtradegym.py why
obs = np.array([ # row.open, # row.high, # row.low, # row.close, # row.volume,

are pulled out of the observation space? Why you consider at least open and close not informative?
edit: At least normalized

in IndicatorforRL.py there are still the buying and selling conditions, are there for compatibility issues or for what? also because in LoadRLmodel.py the buying/selling conditions call the model.zip

i see in freqtradegym
self.stake_amount = self.config['stake_amount']

self.reward_decay = 0.0005 self.not_complete_trade_decay = 0.5 self.game_loss = -0.5 self.game_win = 1.0 self.simulate_length = self.config['gym_parameters']['simulate_length']

the reward parameters, this drives me to ask: Is the profit the objective function?

in config

"gym_parameters": { "indicator_strategy": "IndicatorforRL", "fee": 0.0015, "timerange": "20201120-20201130", "simulate_length": 200 },
is timerange the observation space? If yes simulate_lenght are the Max_Episodes trained ONTO the timerange? Am i right? So regarding yours example in 15mins timeframe are 200 15min timesteps?

Do you mind switch to rllib?

Complete newb. After running, can model.zip immediately be used?

Just wondering if you can give a few sentence crash course on what this actually does?

I've been looking at/messing with the freqtrade bot for a while.

I've had a small introduction into machine learning.

Just wondering, what does this actually do? Is the model.zip something that can be used with a running strategy?

If I am asking these questions, ...will practical use of this not be something within the scope of responding to this issue? :)

Hoping you can provide some insight.

(side note):

What I had hoped that this would do, looking at the code, is look at the different TA methods in IndicatorforRL.py under populate_indicators, and output what the best combo it found was.

Any information you can provide would be much appreciated.

Adding multiple tickers to white pairlist

First of all, thanks for this amazing repository and your great work. I am trying to extend the model of yours If I can manage, I would like to send PR request. For the question:

I trained 2 models. One with just USDT/BTC pair and the other is with 160 different pairs of BTC.
The training duration and average reward in the end was same. I thought training would be much slower and the average reward would be much higher. Then I thought, Is my model really use other pairs which I added to config_rl.json ?

Also is there a way to give the coin's name as an observation so the model may understand the difference between different coins. For example BTC and ETH are much stable then DODGE coin or SHIB so maybe model can act differently on them.

Here is a photo of that 2 training.

which model should I pass as a parameter to ACER

I don't have much experience with RL.

But following the tutorial step by step, I don't know exactly how to fill the model from the file "LoadRLModel.py" on line 29:

model = ACER.load('model')

What should I put in place in 'model'? Would anyone have an example of a template?

neat_config not found on the repo

Hi,

I tried launching the neat_trade.py and had an exception : No such config file: /home/raed/dev/freqtrade_gym/freqtrade/neat_config
Could you please push the configuration file on the repo ? Thanks in advance.

NameError: name 'SubprocVecEnv' is not defined

Hi
Thanks very much for your work. I get an error when I run deep_rl.py. Here I paste it: (I run it on Ubuntu 18.04.5, Python 3.6, Tensorflow 1.14 )
python deep_rl.py
/home/mycomp/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:493: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:494: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:495: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:496: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:497: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:502: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Traceback (most recent call last):
File "deep_rl.py", line 21, in
env = SubprocVecEnv([lambda: TradingEnv(config)]*12)
NameError: name 'SubprocVecEnv' is not defined

Impossible to load Strategy 'IndicatorforRLFull'

Hi
First of all thanks very much for your support. I'm using your updated files and once I run the the code "python deep_rl.py" I get this error. I copied the strategy "IndicatorforRLFull" in freqtrade/user_data/strategies but it seems that it doesn't work. An idea why this happens?
Thanks a million 🥇

File "deep_rl.py", line 33, in
env = TradingEnv(config)
File "/home/frequi/freqtrade/freqtradegym.py", line 36, in init
self.strategy = StrategyResolver.load_strategy(config)
File "/home/frequi/freqtrade/freqtrade/resolvers/strategy_resolver.py", line 47, in load_strategy
extra_dir=config.get('strategy_path'))
File "/home/frequi/freqtrade/freqtrade/resolvers/strategy_resolver.py", line 192, in _load_strategy
f"Impossible to load Strategy '{strategy_name}'. This class does not exist "
freqtrade.exceptions.OperationalException: Impossible to load Strategy 'IndicatorforRLFull'. This class does not exist or contains Python code errors.

Tensorflow Version

Hi,

could it be that this project is based on Tensorflow 1.15? I do get dependency errors when trying to run.
It complain about deepq missing tensorflow.contrib, but that is deprecated in Tensorflow 2.
Would it be possible for you to update the project to Tensorflow 2?

AttributeError: 'Pandas' object has no attribute 'kc_percent'

Hi
I'm here again with a new issue 🔢. When I run deep.rl.py I get this error "AttributeError: 'Pandas' object has no attribute 'kc_percent'". Probabably it's something with indicators but I couldn't figure it out. copy the error message here below FYI. I'd appreciate your effort to create this project.

/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/frequi/freqtrade/.env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
{236429}
Traceback (most recent call last):
File "deep_rl.py", line 29, in
model.learn(total_timesteps=int(1e+6))
File "/home/frequi/freqtrade/.env/lib/python3.6/site-packages/stable_baselines/acer/acer_simple.py", line 578, in learn
enc_obs, obs, actions, rewards, mus, dones, masks = self.runner.run(callback)
File "/home/frequi/freqtrade/.env/lib/python3.6/site-packages/stable_baselines/common/base_class.py", line 794, in runner
self._runner = self._make_runner()
File "/home/frequi/freqtrade/.env/lib/python3.6/site-packages/stable_baselines/acer/acer_simple.py", line 240, in _make_runner
return _Runner(env=self.env, model=self, n_steps=self.n_steps)
File "/home/frequi/freqtrade/.env/lib/python3.6/site-packages/stable_baselines/acer/acer_simple.py", line 682, in init
super(_Runner, self).init(env=env, model=model, n_steps=n_steps)
File "/home/frequi/freqtrade/.env/lib/python3.6/site-packages/stable_baselines/common/runners.py", line 31, in init
self.obs[:] = env.reset()
File "/home/frequi/freqtrade/.env/lib/python3.6/site-packages/stable_baselines/common/vec_env/dummy_vec_env.py", line 61, in reset
obs = self.envs[env_idx].reset()
File "/home/frequi/freqtrade/freqtradegym.py", line 238, in reset
return self._next_observation()
File "/home/frequi/freqtrade/freqtradegym.py", line 112, in _next_observation
row.kc_percent,
AttributeError: 'Pandas' object has no attribute 'kc_percent'

error while running python neat_trade.py

I am getting error while running python neat_trade.py
Traceback (most recent call last):
File "neat_trade.py", line 25, in
from gym_env.trading_env import TradingEnv
ModuleNotFoundError: No module named 'gym_env'

Adding Trade Information to the observation space

Hi I saw that you created the trading_status however you didnt add that to the observation list. Is there a way to add that on to the observation list ?

No results

Hi
After running the RL model I get a zip file called model.zip and a folder called Tensorboard, however, when I open the model.zip the text files have no results and I cannot figure out why this happens, please see below:

When I open parameter_list I see this:
[
"model/pi_fc0/w:0",
"model/pi_fc0/b:0",
"model/vf_fc0/w:0",
"model/vf_fc0/b:0",
"model/pi_fc1/w:0",
"model/pi_fc1/b:0",
"model/vf_fc1/w:0",
"model/vf_fc1/b:0",
"model/vf/w:0",
"model/vf/b:0",
"model/pi/w:0",
"model/pi/b:0",
"model/q/w:0",
"model/q/b:0"
]

and I cannot see the results... is this because of any package incompatibility? I'm using Python 3.6, Tensorflow and tensorboard 1.14.0, numpy 1.19.4, and the most recent freqtrade version. Thanks very much in advance for your help and excellent work :-)

:~/$ tensorboard --logdir ./tensorboard
Traceback (most recent call last):
File "/home/frequi/.local/bin/tensorboard", line 7, in
from tensorboard.main import run_main
File "/home/frequi/.local/lib/python3.6/site-packages/tensorboard/main.py", line 40, in
from tensorboard import default
File "/home/frequi/.local/lib/python3.6/site-packages/tensorboard/default.py", line 38, in
from tensorboard.plugins.audio import audio_plugin
File "/home/frequi/.local/lib/python3.6/site-packages/tensorboard/plugins/audio/audio_plugin.py", line 30, in
from tensorboard.util import tensor_util
File "/home/frequi/.local/lib/python3.6/site-packages/tensorboard/util/tensor_util.py", line 20, in
import numpy as np
File "/home/frequi/.local/lib/python3.6/site-packages/numpy/init.py", line 140, in
from . import core
File "/home/frequi/.local/lib/python3.6/site-packages/numpy/core/init.py", line 68, in
raise ImportError(msg.format(path))
ImportError: Something is wrong with the numpy installation. While importing we detected an older version of numpy in ['/home/frequi/.local/lib/python3.6/site-packages/numpy']. One method of fixing this is to repeatedly uninstall numpy until none is found, then reinstall this version.

AttributeError: 'IndicatorforRL' object has no attribute 'ohlcvdata_to_dataframe'

I'm trying to use it for the first time and it's giving me this error. Do you know if anything has changed in the structure of the project?

C:\Users\marco\anaconda3\envs\rl\lib\site-packages\stable_baselines_init_.py:33: UserWarning: stable-baselines is in maintenance mode, please use Stable-Baselines3 (SB3) for an up-to-date version. You can find a migration guide in SB3 documentation.
"stable-baselines is in maintenance mode, please use Stable-Baselines3 (SB3) for an up-to-date version. You can find a migration guide in SB3 documentation."
Add {
"method": "AgeFilter",
"min_days_listed": 30
},
to your pairlists in config (Under StaticPairList)
Traceback (most recent call last):
File "deep_rl.py", line 14, in
env = TradingEnv(config)
File "C:\Users\marco\trade\freqtrade\freqtradegym.py", line 44, in init
preprocessed = self.strategy.ohlcvdata_to_dataframe(data)
AttributeError: 'IndicatorforRL' object has no attribute 'ohlcvdata_to_dataframe'

Example for RLlib

Hello, can you please provide example of loading model for backtesting and trading with RLlib?

The file model could not be found

Hello, thanks for sharing the project .
I get an error running: python deep_rl.py

raise ValueError("Error: the file {} could not be found".format(load_path)) ValueError: Error: the file model could not be found

Any idea how to fix it?

Thanks

ImportError: DLL load failed while importing MPI:

File "C:\Users\Kit\PycharmProjects\Freqconda\freqtrade\venv\lib\site-packages\stable_baselines\ddpg\ddpg.py", line 12, in
from mpi4py import MPI
ImportError: DLL load failed while importing MPI: Impossibile trovare il modulo specificato.

ValueError: numpy.ndarray size changed, may indicate binary incompatibility.

Hi, thanks for your work. A long time ago I worked about IA and trading, but never found a way to incorporate it too fractured.
I have this error despite I followed your guideline, please could you help me?

(.env) (base) onetwo31@onetwo31-VirtualBox:~/Freqtrade_ia/freqtrade$ python rllib_example.py
Traceback (most recent call last):
File "rllib_example.py", line 6, in
from freqtradegym import TradingEnv
File "/home/onetwo31/Freqtrade_ia/freqtrade/freqtradegym.py", line 12, in
import talib.abstract as ta
File "/home/onetwo31/freqtrade/.env/lib/python3.8/site-packages/talib/init.py", line 52, in
from ._ta_lib import (
File "talib/_ta_lib.pyx", line 1, in init talib._ta_lib
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject