revenyouio / bots-lab Goto Github PK

Backtesting tool and connector for the platform Bots.

Python 29.62% Shell 0.81% Dockerfile 0.47% Batchfile 0.35% Jupyter Notebook 29.83% C# 22.46% R 1.55% MQL5 14.89%

bots-lab's Introduction

Bots Lab Backtesting And Connector

Open Source Backtesting Tool. With this tool you can test your bot with historical data to see how it performs. The results will be compared against the benchmark. See bots/bot_template.py and config_test.py for more information.

This project also allows you to create a bot that runs "live", e.g. a bot that sends signals to buy and sell based on actual data. To run a bot in live mode, run it on a (virtual) server. A Linux-based server is usually the cheapest and most efficient option.

The following guidelines apply for a Linux-based server:

The server should have a recent 64-bit version of Ubuntu Linux as its operating system. Using version 18.04 or 20.04 (when it becomes available) is recommended.
The server should have at least 1 GB of memory and 20 GB of disk space.
You need to have administrative access (via root or sudo) to the server via a terminal (e.g SSH).

If you want to run the bot on a server your own infrastructure contact your systems or network administrator.
You can order suitable virtual machines from the following companies:

DigitalOcean: https://www.digitalocean.com/products/droplets/
Microsoft: https://azure.microsoft.com/en-us/services/virtual-machines/

Once you have a server available, follow the instructions in the first section to allow it to run this project.

This README contains the following:

Instructions to get a development and live environment up and running. See Preparing your development or live environment.
Instructions to create your own bot. See Create and run your own bot.

Preparing your development or live environment

To run an existing bot or to create your own, you will need to prepare your computer to allow this project to run. To do this, follow the instructions in this section.

For instructions to create your own bot, see Create and run your own bot.

Recommended: running the bot using Docker

The easiest way to get up and running is by using Docker. When using Docker, you won't have to install any platform-specific dependencies (e.g. Python). Only Docker itself is required.

If necessary, install Docker.

On Windows and macOS: install Docker Desktop from https://www.docker.com/get-started.

On Ubuntu Linux: install Docker and Docker Compose using the instructions on the following pages:
- https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04
- https://www.digitalocean.com/community/tutorials/how-to-install-docker-compose-on-ubuntu-18-04
These instructions should also apply for servers from other suppliers. A Docker Hub account is not required.
Clone the Bot-Backtesting project to your computer using Git if you haven't already done so.
Open a command prompt, cd to the project directory and run one of the following commands:

Platform Testing mode Live mode

macOS and Linux ./run_bot.sh test ./run_bot.sh live

Windows run_bot.cmd test run_bot.cmd live

Platform	Testing mode	Live mode
macOS and Linux	`./run_bot.sh test`	`./run_bot.sh live`
Windows	`run_bot.cmd test`	`run_bot.cmd live`

Remarks:

When running one of these commands for the first time, a Docker image will be built for the bot. This may take a few minutes and produce a lot of output. Next runs will be faster and quieter.
In testing mode, the bot runs once and exits automatically. Output is shown immediately.
In live mode, the bot runs periodically in the background. By default the bot runs every 30 minutes. Also, the bot will start automatically when Docker starts. To show output from the bot, run docker logs docker_bot_1.
To stop the bot in live mode, run ./run_bot.sh stop_live or run_bot.cmd stop_live.
You can add extra dependecies by adding them to the requirements.txt file. See the remark below.
After changing any of the files (see Create and run your own bot below), run the test command from the table above again to test your changes. To update the live version with your changes, run the stop_live command and start the live version again.

Alternative: running the bot without Docker

If Docker is unavailable for your platform, or if you prefer not to use it, follow the instructions below to manually install the dependencies required to run a bot.

Download and install Python 3

For this project Python 3 is needed. Mac users take note: Python is already installed as part of your operating system, but this will not suffice. For downloading Python 3 see Download Python.

Install library TA-Lib

TA-Lib is widely used by trading software developers for perform technical analysis of financial market data. Before you can use TA-Lib from Python you must first install the underlying TA-Lib library.

To install the underlying dependency TA-lib:

On Mac OS: install TA-lib using Brew: brew install ta-lib
On Linux, follow the instructions on installing TA-Lib from source.
On Windows, download prebuilt Python packages (wheels) of TA-Lib for your Python version from https://www.lfd.uci.edu/~gohlke/pythonlibs/#ta-lib and place them in the wheels folder.

For more information:

Install Python dependencies

On Linux and macOS: run bash create_venv.sh from a terminal. On Windows run create_venv.cmd.

These scripts will create a virtual Python environment in the directory venv below the project directory. This virtualenv contains all required Python dependencies. They are listed in requirements.txt. After adding extra dependencies to this file you must run bash create_venv.sh or create_venv.cmd again.

Running the bot in test mode

On macOS and Linux: venv/bin/python3 run_test.py

On Windows: venv\scripts\python3 run_test.py

Running the bot in live mode

To run the bot in live mode, you will need to schedule a task that periodically runs your bot. This can be done using cron on macOS and Linux:

Run the following command to start editing the cron file of the current user: crontab -e

Add the following lines, replacing /path/to/project with the path to the project:

# Run bot every 30 minutes:
*/30 * * * *   /path/to/project/venv/bin/python3 /path/to/project/run_live.py

Save the file and exit the editor.

For a basic introduction to use the most common editors available on macOS and Linux, see this article: https://www.linux.com/topic/desktop/introduction-text-editors-get-know-nano-and-vim/

Create and run your own bot

This project uses by default a simple bot based on the RSI (Relative Strength Index) algorithm. You can find this bot in bots/rsi.py. This bots folder also contains ema.py and sma.py, two bots that use the Ta-Lib library. To create your own bot, follow the instructions in this section of the README.

Writing your first bot

See bots/bot_template.py for an explanation about writing your first bot. This means you have to implement the function get_buy_or_sell_signal and store that function in a python file (extension .py) in the bots folder.

Configuration Backtesting tool

See config_test.py for the simulation parameters of the Backtesting tool and the data settings for the historical data.

Run your bot against the backtesting engine

Steps:

In config_test.py give the parameter bot_name the right value: bot_name = 'bots.[file_name_of_bot]'
Execute the bot in testing mode using the instructions above, e.g. with the recommended Docker command: run_bot.sh test (or run_bot.cmd test on Windows).

Run your bot live

Steps:

In config_live.py give the parameter bot_name the right value: bot_name = 'bots.[file_name_of_bot]'
Execute the bot in live mode using the instructions above, e.g. with the recommended Docker command: run_bot.sh live (or run_bot.cmd live on Windows).

bots-lab's People

Contributors

Stargazers

Watchers

bots-lab's Issues

Allow variable storage

Currently it's not possible to store any variables. For example you want to check a variable multiple ticks before taking action, or you use an indicator to define a threshold that you want stored. (Maybe this aligns with the store current position ticket, depending on how it is designed)

Allow Multi-Pair trading

Currently you have to select one pair to trade one. It would be nice if you can select multiple pairs to run at the same time (and split funds by choice on these pairs). To consider:

If they don't have the same base currency, total has to be calculated (in one converted currency?). For now we always choose the same base currency.
You have to be able to check the status of every trade, and adjust take profit and stop loss per currency. This is a nice to have for in the future.
You have to be able to select how many funds are used in the trade
Possibly select margin percentage per pair/trade too. This is a nice to have for in the future.

generate csv or excel

It would be great if next to the backtesting graph a file with the entry point (time, price, direction) and subsequently exit point (time, price, direction) would be created, so we can analyse it more in depth.

analyze() function in youengine.py throws error(s)

Function throws errors in:

 if(analyze):
             self.analyze(title_suffix=self.pair, **kwargs)

Fixing this is a distraction as the analyze function is a placeholder and doesn't do anything - should be removed?

def analyze(self):
        pass

Docker can't find bot strategies

After building the docker and trying to read an other strategy (the default ones or any self made strategy) the error 'bot not found' keeps coming up. Latest terminal output below:

Successfully tagged docker_bot:latest
Recreating docker_bot_1 ... done
Attaching to docker_bot_1
bot_1  | /usr/local/lib/python3.8/site-packages/pandas_datareader/compat/__init__.py:7: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
bot_1  |   from pandas.util.testing import assert_frame_equal
bot_1  | Traceback (most recent call last):
bot_1  |   File "run_test.py", line 10, in import_bot
bot_1  |     return importlib.import_module(name)
bot_1  |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
bot_1  |     return _bootstrap._gcd_import(name[level:], package, level)
bot_1  |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
bot_1  |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
bot_1  |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
bot_1  |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
bot_1  |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
bot_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
bot_1  |   File "/opt/bot/bots/sma.py", line 1, in <module>
bot_1  |     import talib
bot_1  |   File "/usr/local/lib/python3.8/site-packages/talib/__init__.py", line 43, in <module>
bot_1  |     from ._ta_lib import (
bot_1  | ImportError: libta_lib.so.0: cannot open shared object file: No such file or directory
bot_1  | 
bot_1  | During handling of the above exception, another exception occurred:
bot_1  | 
bot_1  | Traceback (most recent call last):
bot_1  |   File "run_test.py", line 29, in <module>
bot_1  |     bot = import_bot(name = bot_name)
bot_1  |   File "run_test.py", line 12, in import_bot
bot_1  |     raise Exception(f'Bot module {name} does not exist')
bot_1  | Exception: Bot module bots.sma does not exist
docker_bot_1 exited with code 1

Bot opens new buy while in buy position

When setting up a test strategy I defined the following strategy (where both sell and buy have the same requirements to force buys only)

macd, macdsignal, macdhist = talib.MACD(data['close'].values, fastperiod=12, slowperiod=26, signalperiod=9)
    if macdhist[-1] > 0:
        return 'buy'
    elif macdhist[-1] > 0:
        return 'sell'
    else:
        return None

The result yields a lot of buys, while the bot is only able to do one buy (since after that all funds are used).

TypeError: 'type' object is not subscriptable

I have a error.
My strategy is build in the freqtrade style so now i want to transform it to the bots style, but is a little complex. I found it difficult because bots cannot accept libary's used by freqtrade so you need to rewrite your whole strat, Now do i have a error whom i found it hard to find a answer online for so i ask directly to you. In the following code do i get this erorr:

from ctypes.wintypes import SHORT
from fileinput import close
from operator import truediv
import numpy as np # noqa
import pandas as pd # noqa
from functools import reduce
from pandas import DataFrame
import talib

--- Add your lib to import here ---

SHORT = 56
LONG = 32

rsi = talib.RSI

upperbb1, middlebb1, lowerbb1 = talib.BBANDS(DataFrame['close'], nbdevup=1, nbdevdn=1, timeperiod=20)
upperbb2, middlebb2, lowerbb2 = talib.BBANDS(DataFrame['close'], nbdevup=2, nbdevdn=2, timeperiod=20)
upperbb3, middlebb3, lowerbb3 = talib.BBANDS(DataFrame['close'], nbdevup=3, nbdevdn=3, timeperiod=20)
upperbb4, middlebb4, lowerbb4 = talib.BBANDS(DataFrame['close'], nbdevup=4, nbdevdn=4, timeperiod=20)

def get_buy_or_sell_signal(data):
"""
Main algorithm method (the actual bot!), which will be called every tick.
This method receives a Pandas DataFrame(data). Each row of this Dataframe is a candle.
Rows of a DataFrame are dictionary like objects. To access a row value you can use
the follwing syntax: row_name['column_name']. Based upon this DataFrame the bot must
make a decision to buy, sell or do nothing. See below for a simple example.

Param data: 
List of candles or ticker data in the form of a Pandas Dataframe. The test environment
uses candles (historical data) and the live environment ticker data. Candles or ticker data 
have minimal the following properties: date, open, high, low, close and volume.

Float64:
The Dataframe number values are of type float64 (numpy dtype float). To convert this values to native
Python floats use the item() function. See below.

Example candle / ticker data:
date(index)                 open       high       low        close       volume      date
2020-03-03 12:00:00     0.02608580 0.02613449 0.02581001 0.02581078  59.40688146 2020-03-03 12:00:00

Return: 'buy', 'sell' or None
"""



# skip when the number of rows (candles) is too short
if len(data) < 2:
    return None

short = rsi(data['close'].values) 
            
        

long = rsi(data['close'].values)

# get the last and second to last row
current_candle = data.iloc[-1]
previous_candle = data.iloc[-2]

# return sell signal
if data['close'] < lowerbb1 and  long[-1]> 32 :
    return 'buy'
elif data['close'] > middlebb1 and short[-1]>56:
    return 'sell'
else:
    return None

The error is get is this one:
Traceback (most recent call last):
bot_1 | File "run_test.py", line 41, in
bot_1 | bot = import_bot(name = bot_name)
bot_1 | File "run_test.py", line 10, in import_bot
bot_1 | return importlib.import_module(name)
bot_1 | File "/usr/local/lib/python3.8/importlib/init.py", line 127, in import_module
bot_1 | return _bootstrap._gcd_import(name[level:], package, level)
bot_1 | File "", line 1014, in _gcd_import
bot_1 | File "", line 991, in _find_and_load
bot_1 | File "", line 975, in _find_and_load_unlocked
bot_1 | File "", line 671, in _load_unlocked
bot_1 | File "", line 843, in exec_module
bot_1 | File "", line 219, in _call_with_frames_removed
bot_1 | File "/opt/bot/bots/bbrsi.py", line 19, in
bot_1 | upperbb1, middlebb1, lowerbb1 = talib.BBANDS(DataFrame['close'], nbdevup=1, nbdevdn=1, timeperiod=20)
bot_1 | TypeError: 'type' object is not subscriptable
docker_bot_1 exited with code 1

What do i need to change in order to let it work? I would appreciate it if you could answer me.

Dataframe values are float64 values

Currently the dataframe values are float64 values I believe, I was wondering if it is possible to turn these values to native floats so that they don't need to be converted or if there is a default way to deal with this so that there is no time spent on fixing this.

Allow stoploss/takeprofit variables position and tick based

When in a buy position, I'd like to define my stoploss and take profit every tick and update them accordingly. Currently it's not possible to set these based on the current position, and it's not possible to adjust these variables in the buy_and_sell script every tick.

Latest version of jinja2 is breaking Bots-Lab

Due to changes in the latest released jinja2 the project is failing to run.
The following exception is thrown:

Attaching to docker_bot_1
bot_1  | Traceback (most recent call last):
bot_1  |   File "run_test.py", line 4, in <module>
bot_1  |     from youengine.helpers.analyze import analyze_mpl, analyze_bokeh
bot_1  |   File "/opt/bot/youengine/helpers/analyze.py", line 4, in <module>
bot_1  |     import bokeh.plotting
bot_1  |   File "/usr/local/lib/python3.8/site-packages/bokeh/plotting/__init__.py", line 67, in <module>
bot_1  |     from ..document import Document; Document
bot_1  |   File "/usr/local/lib/python3.8/site-packages/bokeh/document/__init__.py", line 35, in <module>
bot_1  |     from .document import DEFAULT_TITLE ; DEFAULT_TITLE
bot_1  |   File "/usr/local/lib/python3.8/site-packages/bokeh/document/document.py", line 49, in <module>
bot_1  |     from ..core.templates import FILE
bot_1  |   File "/usr/local/lib/python3.8/site-packages/bokeh/core/templates.py", line 42, in <module>
bot_1  |     from jinja2 import Environment, FileSystemLoader, Markup
bot_1  | ImportError: cannot import name 'Markup' from 'jinja2' (/usr/local/lib/python3.8/site-packages/jinja2/__init__.py)
docker_bot_1 exited with code 1

By adding a specific version range in requirements.txt for jinja2, the issue is fixed.

Jinja2>=2.10.1,<3.1

PullRequest: #26