debut-js / core Goto Github PK

Constructor of trading strategies. Based on a flexible core and plugin system. It allows you to configure strategies from ready-made components, only by setting the points of entry into the market. Debut also allows backtesting and genetic optimizations.

Home Page: https://debutjs.io

License: GNU General Public License v3.0

JavaScript 0.76% TypeScript 99.24%

core's Introduction

Debut - Trading Framework

Debut is an ecosystem for developing and launching trading strategies. An analogue of the well-known ZenBot, but with much more flexible possibilities for constructing strategies. All you need to do is come up with and describe the entry points to the market and connect the necessary plugins to work. Everything else is a matter of technology: genetic algorithms - will help you choose the most effective parameters for the strategy (period, stops, and others), ticker selection module - will help you find an asset suitable for the strategy (token or share), on which it will work best.

Debut is based on the architecture of the core and add-on plugins that allow you to flexibly customize any solution. The main goal of the entire Debut ecosystem is to simplify the process of creating and launching working trading robots on various exchanges.

Features

Multiple exchanges API
Backtesting with historical data
Backtesting results visualization
Backtesting live preview
Strategy optimisation (genetic algorithms, multi thread)
Stretegy overfitting control (Walk-Forward)
Cross timeframe candles access
Simple working with data using callbacks e.g. onCandle, onTick, onDepth ...
Written in TypeScript (JavaScript), may be executed in browser
Customizable with plugins
Can use community edition for free with limitations

Available brokers

Didn't see your broker? You can donate for support.

Community edition

We believe in the power of the community! That is why we decided to publish the project. The community version is free, but it has some limitations in commercial use (income from trading startups is not commerce), as well as technical differences in testing strategies. Join the community, join the developer chat

Enterprise edition

(Available by subscription for $20/mo)

Cross timeframe candles access (from lower to higher candles)
Advanced tick emulation in backtesting (60+ ticks per candle)
Tinkoff and Alpaca supports all timeframes (programmaticaly solved broker issue)

We are streaming Enterprise-based deals live on our telegram channel

Find out the price by sending a request to [email protected]

Remember! Starting a bot and trading in general requires careful study of the associated risks and parameters. Incorrect settings can cause serious financial losses.

The project has two starting trading strategies "For example" how to work with the system.

CCI Dynamic strategy report for last 100 days

Report analysis

startBalance: 500
balance: 1183.97
maxBalance: 1186.37
minBalance: 500
maxMarginUsage: 1892.54
profit: 683.97
long: 273
longRight: 243
short: 282
shortRight: 277
absoluteDD: 21.06
relativeDD: 30.31
potentialDD: 51.9
maxWin: 6.21
maxLoose: -8.14
profitProb: 0.94
looseProb: 0.06
avgProfit: 4.54
avgLoose: 47.95
expectation: 1.23
failLine: 6
rightLine: 20
avgFailLine: 1.93
avgRightLine: 5.36
ticksHandled: 9637
candlesHandled: 9636

Strategy statistics were collected based on the plugin statistics, follow the link to learn more about the meaning of some statistics.

Visualization is done using the Report plugin.

System requirements

To work, you need NodeJS 16.xx/npm 7.xx (installation instructions)

Documentation

Project file structure

| - .tokens.json - custom access tokens for working with the exchange
| - schema.json - description of the location of the startup files
| - public/- folder for finder reports (created when finder starts)
| - src/
    | - strategies/
        | - strategy1/- strategies directory
            | - bot.ts - Strategy implementation
            | - meta.ts - Meta data, for launching and for optimization
            | - cfgs.ts - Configurations, for launching in tester and genetic
        | - strategy2/
        ...

Installation and configuration

Obtaining API tokens

Installing tokens

To work, you need to create a .tokens.json file, add a token to it for work.

For Tinkoff (instructions):

{
    "tinkoff": "YOUR_TOKEN",
    "tinkoffAccountId": "YOUR_ACCOUNT_ID",
}

Get your tinkoff accounts by this hack and copy one of available account identity to tinkoffAccountId:

execute command npm run getAccountId

tinkoffTransport.api.users.getAccounts({}).then(res => {
    console.log(res);
});

For Binance (instructions):

{
    "binance": "YOUR_TOKEN",
    "binanceSecret": "YOUR_SECRET"
}

For Alpaca (Alpaca instructions):

{
    "alpacaKey": "YOUR_TOKEN",
    "alpacaSecret": "YOUR_SECRET"
}

All tokens can be getted by call: cli.getTokens() also you can use any field name for the token.

Installing npm packages

To install packages, run:

npm install

Creating strategy

Command will create strategy in src/strategies direction with bot.ts, cfgs.ts and meta.ts files inside and add it to schema.json

npm run plop StrategyName

Build the project

npm run compile

It is recommended to build before each test run

npm run compile && npm run ...

Start testing on historical data

Historical data will be loaded automatically at startup All loaded data is saved in the history folder in the root of the project, then reused.

Before starting, make sure:

The cfgs.ts file contains the ticker you need
To get history in the.tokens.json filea token may be required
The history of a stock or token exists in the requested number of days

To start, run the command:

npm run testing -- --bot=FTBot --ticker=CRVUSDT --days=200 --gap=20

To view the test results in a browser, execute

npm run serve

The results will be available for viewing on http://localhost: 5000/

You can read more about the test run parameters in the documentation

Run genetic optimization

Run the command:

npm run genetic -- --bot=FTBot --ticker=CRVUSDT --days=200 --gap=30 --gen=12 --pop=2000

or with gap 60 days from now, and WFO enabled with type rolling also available --wfo default type is 'anchored' wfo

npm run genetic -- --ticker=BTCUSDT --bot=FTBot --amount=500 --gen=25 --pop=10000 --days=300 --gap=60 --wfo=rolling

More details about the launch parameters of the genetics can be found in the documentation

After starting with the --log parameter, the geneticist will output data to the console

Binance history loading from Wed Nov 18 2020 03:00:00 GMT + 0300 (Moscow Standard Time) ...

----- Genetic Start with 17314 candles -----

Generation: 0
Generation time: 5.15 s
Stats: {
   population: 100,
   maximum: 20.8,
   minimum: -1.24,
   mean: 2.5174,
   stdev: 3.8101996325652054
}
Generation: 1
...

Run genetic optimization with tickers/tokens selection

Run the command:

npm run finder -- --bot=FTBot --ticker=CRVUSDT --days=200 --gap=30 --gen=12 --pop=2000 --log

Use the --crypt option to take crypto pairs from the./Crypt.json file (By default, there are actual Binance cross-margin pairs)

By default, a set of stock tickers is used for streaming optimization from the file stocks.json

You can read more about the parameters for launching streaming genetics in the documentation

Starting a strategy

Install any process manager for NodeJS, for example PM2,,,

npm install -g pm2

Execute the launch command, the path to the strategy launch file in the ./Out directory. An example of such a file can be found here /src/bootstrap.ts

pm2 start ./out/bootstrap.js

Further, for operation and monitoring, you can use pm2 command:

pm2 list - a list of active processes

pm2 delete $ pid - stop a process

pm2 log - to view the logs of running processes

and other commands, which can be found in the documentation of the process manager.

Debut is flexible for any tools

Buy / Sell ratio

A measure of the activity rating of buyers and sellers for several pairs. This tool is based on the method of counting all purchases listed on the exchange and all sales, they are calculated for several large tokens dominating the market. Parameter tickers = ['ETH', 'BTC', 'ETC', 'XRP']; A fast and slow moving average (SMA) is used to smooth and track changes in activity. Further, for less voluminous tokens (altcoins), the correlation between the change in their price and the change in the buyer's index is calculated. Deals are formed for a certain coin according to the simple logic that if the buyer's activity is above 50% and the price correlation is high, or the share of buyers is less than 50% and the price correlation is high (falling).

Start

npm run compile && node ./out/tools/buysellratio/index.js

Screenshot

core's People

Contributors

Stargazers

Watchers

core's Issues

Proposal for market depth snapshot

Requirements

For using onDepth hook we need to get full depth snapshot, because onDepth stream using delta updates for price and volume at this price in current time.

Transport snapshot api call

getDepthSnapshot(): Map<number, number> // method returns object with price and volume at this price

Core hook

// Core method noop (implementation in strategy required by developer)
protected async onDepth(depth: Depth): Promise<void> {}

Strategy hook

async onDepthSnapshot(snapshot) {
   this.orderBookAggregator.useSnapshot(snapshot);
}

Market Depth builder example

class OrderBookAggregator {
    private bids = new Map<number, number>();
    private asks = new Map<number, number>();
    private splitPrice = 0;
    private maxPercent = 10;
    private precision = 1;
    private round = 50;

    constructor() {}

    update(depth: Depth) {
        const bidsSize = depth.bids.length - 1;

        depth.bids.forEach((item, idx) => {
            this.bids.set(item.price, item.qty);

            if (idx === bidsSize) {
                this.splitPrice = item.price;
            }
        });

        depth.asks.forEach((item) => {
            this.asks.set(item.price, item.qty);
        });

        // Keep memory clean
        this.bids.forEach((qty, price) => {
            const percent = math.percentChange(this.splitPrice, price);

            if (percent > this.maxPercent || percent < -this.maxPercent || price > this.splitPrice) {
                this.bids.delete(price);
            }
        });

        this.asks.forEach((qty, price) => {
            const percent = math.percentChange(this.splitPrice, price);

            if (percent > this.maxPercent || percent < -this.maxPercent || price <= this.splitPrice) {
                this.asks.delete(price);
            }
        });

        console.clear();

        this.printAsks();
        console.log('--------');
        this.printBids();
        // console.log(this.asks);
    }

    printBids() {
        console.log('Buyers (BID):');
        const result = {};
        Array.from(this.bids.keys()).forEach((key) => {
            const transformedKey = this.getKey(key);
            result[transformedKey] = result[transformedKey] || 0 + this.bids.get(key);
        });

        Object.keys(result)
            .sort()
            .reverse()
            .slice(0, 15)
            .forEach((key) => {
                console.log(key, result[key]);
            });
    }

    printAsks() {
        console.log('Sellers (ASK):');
        const result = {};
        Array.from(this.asks.keys()).forEach((key) => {
            const transformedKey = this.getKey(key);
            result[transformedKey] = result[transformedKey] || 0 + this.asks.get(key);
        });

        Object.keys(result)
            .sort()
            .reverse()
            .slice(-15)
            .forEach((key) => {
                console.log(key, result[key]);
            });

        // console.log(sorted);
    }

    private getKey(key: number) {
        if (this.round) {
            key = ~~(key / this.round) * this.round;
        }

        return math.toFixed(key, this.precision);
    }
}

Proposal for partial close order

Requirements

For many strategies good practice close 75% of order when take reached and enable trailing stop for 25% left order.

Examples

this.closeOrder(order: ExecutedOrder, partial: number = 0..1);

this.partialClose(order: ExecutedOrder, partial: number = 0..1);

ExecutedOrder should have flag partial and percentage of closing after that

Add retries when creating orders with Tinkoff API v2

See #20 (comment)

We are in searching for orderbook (depth) stop loss detection.

We need help with algorithm in JavaScript/TypeScript for grouping orderbook incoming data and getting nearest stops one or many. If you have any ideas, suggest it here.

Proposal for runtime snapshot saving / restoring

Requirements:

Debut strategy instance should have a method for getting current runtime state e.g. plugins state calculated at work and current orders.

Plugins and Types

New hooks PluginHook.onSnapshot and PluginHook.onRestore, must return plugin runtime calculated value for savings

Plugin Driver implementation example

	public async getSnapshot() {
		const snapshot: unknown = {};

		for (const plugin of this.plugins) {
			if (PluginHook.onSnapshot in plugin) {
				snapshot[plugin.name] = plugin.onSnapshot();
			}
		}

		return Object.freeze(snapshot);
	}
    
	public async restoreFromSnapshot(snapshot: Record<string, unknown) {
		for (const plugin of this.plugins) {
			if (PluginHook.onRestore in plugin && plugin.name in snapshot) {
				plugin.onRestore(snapshot[plugin.name]);
			}
		}
	}

Core implementations example

public async createSnapshot() {
        const pluginsSnapshot =  await this.pluginDriver.asyncReduce(PluginHook.onSnapshot);
        
        return { orders: this.orders, marketTick: this.marketTick, plugins: this.pluginDriver.getSnapshot() };
}

public async restoreFromSnapshot(snapshot: Record<string, unknown>) {
       return this.pluginDriver.restoreFromSnapshot(snapshot);
}

Strategy implementations example

class MyStrategy extends Debut {
    constructor(...) {
		// Restore saved snapshot
		const snapshot = fs.readFileSync('./snapshot.json', 'utf-8');
		if (snapshot) {
			this.restoreFromSnapshot(JSON.parse(snapshot)):
		}
			
		// Scheduled runtime snapshot savings
		setInterval(this._saveSnapshot, 1000);
    }
    
    private _saveSnapshot() {
		const snapshot = this.createSnapshot();
        
        fs.saveFileSync('./snapshot.json', snapshot);
    }
}

Testing AAPL Alpaca w/StochMacd gives errors; TSLA works

> tester -- "--ticker=AAPL" "--bot=StochMacd" "--days=20"

TypeError: Cannot read property 'broker' of undefined
    at test (Strategies\node_modules\@debut\community-core\lib\cli\tester.js:37:84)
    at Strategies\node_modules\@debut\community-core\lib\cli\tester.js:33:11
    at Object.<anonymous> (Strategies\node_modules\@debut\community-core\lib\cli\tester.js:34:3)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
    at Module.load (internal/modules/cjs/loader.js:937:32)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
    at internal/main/run_main_module.js:17:47

Process finished with exit code 0

For some funny reason, using --ticker=TSLA works.

Cross Validation

Support the cross validation during genetic optimisation process.

Cross-validation is a resampling procedure used to evaluate machine learning models on a limited data sample.

That should prevent overfitting genetical phenotypes.

Fix linting

I've clone your repo and wanted to create a PR on a branch in order to fix the linting, but I can't :(
Later on I want to add a new type of transport for XTB.

Do I need to fork and only after make a PR into your repo ?

Alpaca.markets driver

Nice to have for US users only

FTX driver

It would be awesome to develop a driver for the FTX

createOrder doesn't push order to this.orders until it is executed

I had this trouble with onTick, i has a condition that refers to this.orders.length

if (tick.h > this.maShort1 && this.prevTick.h < this.maShort1 && this.orders.length < 1) { await this.createOrder(OrderType.SELL); }

it can create order on every tick, until the first order is executed, personally i had 7 orders instead of 1

Error codes

All error codes should look more human readable.

High timeframes access hooks

Add hooks to handle h1 h4 and day candles

Proposal: Walk forward 2 types

Problem

The GA (genetical algorithms) model can be overfitted. To prevent overfitting we can use some validations.

Debut core already have Rolling Walk optimization, configured by --walkFwd=aggressive and --walkFwd=conservative

This is full description of two types of optimisations. And we need research and select better.

Debut cli should provide a toggler for first one and another type of WFO. Like --wfoType=rolling and --wfoType=anchored

Abstract

If you take the above example but make the optimisation window bigger and bigger you arrive at an anchored WFA. The optimisation is anchored since it always starts at the same point of time (Figure 6.2A). This is the simplest form of WFA. It usually works best, like the static optimisation we examined above, when the tested market keeps its personality within the tested sample.

Figure 6.2A: Anchored walk forward analysis. For every new optimisation run the in-sample optimisation window starts at the same point of time but its length is subsequently increased. The unseen out-of-sample data follows.

However, as you know, markets and their personalities can be complex. Sometimes a market will show an equity curve that has some very profitable and some not-very-profitable intervals within the tested data ranges. If a market appears to have such a changing personality then the anchored WFA may be not the best method. In these cases it is necessary to perform an optimisation method which better adapts to changing market conditions, the rolling WFA (Figure 6.2B). The rolling WFA is particularly appropriate for short-term intraday systems.

Figure 6.2B: Rolling walk forward analysis. Shifting in-sample optimisation window = 1 year and shifting unseen out-of-sample data = the 3 following months.

The better adaptation of the rolling WFA may mean it is more appropriate, and may yield results that walk forward better, in live trading than results obtained from an anchored WFA.

Continue reading here: The meaning of sample size and market structure

Infinite loop on await this.closeAll

i have this condition called on every tick, it crashes with heap out of memory, it might be caused by this issue ##11
if ( (tick.c > this.maValue && this.prevTick.c < this.maValue) || (tick.c < this.maValue && this.prevTick.c > this.maValue) ) { console.log('Closing all orders'); await this.closeAll(); } )