nautechsystems / nautilus_trader Goto Github PK
View Code? Open in Web Editor NEWA high-performance algorithmic trading platform and event-driven backtester
Home Page: https://nautilustrader.io
License: GNU Lesser General Public License v3.0
A high-performance algorithmic trading platform and event-driven backtester
Home Page: https://nautilustrader.io
License: GNU Lesser General Public License v3.0
The calculation of the current equity and derivatives is based on the executed orders. If this is a high-frequency trading, then everything is ok, but if the period of holding the asset is long, then the volatility of the asset during the holding period is not taken into account. Accordingly, the risk profile may be worse than it shows
perhaps this is a rare case, but nevertheless, users need to know this ...
engine.trader.generate_account_report(BINANCE)
2021-02-01 00:01:03.548456+00:00 | 100000.00000000
balance_USDT
timestamp
2021-02-01 00:01:03.548456+00:00 100000.00000000
2021-02-01 00:02:03.601181+00:00 99933.84172000
2021-02-01 10:02:24.761131+00:00 102448.44010000
2021-02-01 10:03:24.769715+00:00 102379.72808000
2021-02-01 20:03:48.019903+00:00 101407.34064000
... ...
2021-03-16 10:27:17.481950+00:00 139150.87093000
2021-03-16 10:28:17.493945+00:00 139095.26471000
2021-03-16 20:28:36.535599+00:00 138380.19969000
2021-03-16 20:29:36.553532+00:00 138323.88780000
2021-03-16 23:59:43.809309+00:00 137616.88587000
A Betfair integration is currently under development to extend the platforms capabilities into sports betting.
Awesome project! Thanks for open source.
Please specify better way to implement this, to maintain a clean project organization
I am going to use nautilus_trader for backtest and research on Windows and Linux
(and partially in production ?)
Based on your discussion (issue: Windows support #175) and my research and experiments of nautilus_trader code
I suggest the next approach to 'solve' this problem.
1) Windows = Windows 10 64bits (not 32bits) [(Version 2004 OS Build 19041.867)]
Notes:
- base OS on my PC
2) WSL2 = Windows Subsystem for Linux 2
3) Ubuntu = Ubuntu v20.04 on top of Windows (WSL2)
Notes:
- used to support Redis (for NTV_ORIG_Ubuntu, NTV_CORR_Ubuntu and NTV_CORR_Windows!)
- used for testing NTV_ORIG_Ubuntu and NTV_CORR_Ubuntu
4) Redis = redis-server v5.0.7 on Ubuntu
5) Python = CPython 3.8.3 on Windows, CPython 3.8.5 on Ubuntu
6) MSVC = Microsoft Visual C++ 14.0 with 'rc.exe' (on Windows)
Notes:
- used to build NTV_CORR_Windows
To compile (on Windows) *.c files (generated from *.pyx) Cython requires Microsoft Visual C++ 14.0.
7) NTV_ORIG = nautilus_trader original version 0.111.0
8) NTV_CORR = nautilus_trader original version 0.111.0 with changes made
9) NTV_ORIG_Ubuntu = NTV_ORIG building/running/testing on Ubuntu
10) NTV_CORR_Ubuntu = NTV_CORR building/running/testing on Ubuntu
11) NTV_CORR_Windows = NTV_CORR building/running/testing on Windows with Redis (on Ubuntu)
1) uint128_t
====================
Problem description:
--------------------
Really MSVC does not support uint128_t (__uint128_t and any uint128 struct)!
Possible solutions:
--------------------
- I have tryed to code uint128_t in Cython ('C') with low-level operations and such functions
as from_bytes, to_bytes ... It solves the compatibility problem but original code in
nautilus_trsder is faster !?!.
- Proposed solution:
--------------------
Practically nothing do = Not use uint128 at all!
In \core\uuid.pxd
Remove code in NTV_ORIG:
cdef extern from *:
ctypedef unsigned long long uint128 "__uint128_t"
Change code in NTV_ORIG:
cdef readonly uint128 int_val
to code in NTV_CORR:
cdef readonly object int_val
(int_val will store an Python int object)
Version NTV_CORR_Ubuntu gives even better performance results than NTV_ORIG_Ubuntu
(tests/performance_tests/test_perf_uuid.py)
Notes: at these moment possible advantages of C-struct uint128_t is not used in code at all.
Files, classes:
--------------------
\core\uuid.pxd, \core\uuid.pyx (class UUID)
2) PRNG functions
====================
Problem description:
--------------------
MSVC does not contain functions drand48 and srand48 in "stdlib.h"!
Proposed solution:
--------------------
In \backtest\models.pyx
Change code in NTV_ORIG:
cdef extern from "stdlib.h":
double drand48() # Returns a double in range [0,1)
void srand48(long int seedval)
to code in NTV_CORR:
cdef extern from "stdlib.h":
void srand(unsigned int seedval)
int rand()
cdef inline void srand48(unsigned int seedval) nogil:
srand(seedval)
# Returns a double in range [0,1)
DEF RAND_MAX = 32767
cdef inline double drand48() nogil:
return <double>(rand()) / RAND_MAX
Files, classes:
--------------------
\backtest\models.pyx
3) long type
====================
Problem description:
--------------------
With gcc compiler:
On 64-bit architectures, long is at least an int64_t (8 bytes).
On 32-bit architectures, long is at least an int32_t (4 bytes).
With Microsoft compilers:
long is always an int32_t (4 bytes), regardless of an architecture (32/64-bit).
Microsoft compilers support 'long long' (8 bytes) .. types but it's too long long ...
An independence from size (e.g. long) allows to support the portable code.
One of the best way to do this - using the int64_t type.
Proposed solution:
--------------------
Use int64_t type instead of long type.
To import int64_t use:
from libc.stdint cimport int64_t
Files, classes:
--------------------
1) 'timestamp' cases:
Examples:
in core/datetime.pxd
cpdef int64_t to_unix_time_ms(datetime timestamp) except *
cpdef datetime from_unix_time_ms(int64_t timestamp)
...
in core/time.pxd
ctypedef int64_t _PyTime_t
...
in order/order_book.pxd
cdef readonly int64_t update_id
cdef readonly int64_t timestamp
...
2) 'counts' cases: counts for total msec, microsec ...
Examples:
in execution/cache.pyx
cdef int64_t total_us = round((time.time() - ts) * 1000000)
...
in execution/engine.pyx
cdef int64_t total_us = round((self._clock.unix_time() - ts) * 1000000)
...
3) 'get_size_of' cases:
Examples:
in core/functions.pxd
cdef int64_t get_size_of(obj) except *
...
in backtest/data_producer.pxd
cdef int64_t total_size = 0
...
4) converting string to int64_t cases:
Examples:
in model/bar.pyx
from_unix_time_ms(<int64_t>int(pieces[5]))
...
in serialization/parsing.pyx
return None if time_string == NONE else from_unix_time_ms(<int64_t>int(time_string))
...
5) 'thread_id' cases:
Examples:
in common/logging.pxd
cdef readonly int64_t thread_id
in common/logging.pyx
def __init__(
self,
datetime timestamp not None,
LogLevel level,
LogColor color,
str text not None,
int64_t thread_id=0,
):
1) Redis
====================
I don't know nothing about official support for Windows.
But there is official description how to use Redis by Windows applications
(installing Redis on Ubuntu v20.04 on top of Windows (WSL2))
Solution (20-50 min):
--------------------
To run/test NTV_CORR on Windows and use execution DB we have to setup WSL2 on
Windows then Ubuntu and Redis.
For Redis we use default port.
2) uvloop
====================
uvloop team want (wanted) to support Windows since 2016 ...
In nautilus_trader uvloop is not mandatory.
Solution (0 min):
--------------------
Nothing do. Nautilus_trader would just use asyncio with lower performance.
3) MSVC compiler
====================
To compile (on Windows) *.c files (generated from *.pyx) Cython requires Microsoft Visual C++ 14.0.
see "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
Note:
Microsoft Visual Studio 2015 is the compiler used to build the official Python releases for Windows !!!
We also have to download from somewhere (e.g. Windows SDK) rc.exe and rcdll.dll (MS resource compiler)
and copy it to ..\Microsoft Visual Studio 14.0\VC\bin\x86_amd64\
Solution (5-20 min):
--------------------
Load and install
Each test in \tests directory I ran individually for each version
- NTV_ORIG_Ubuntu
- NTV_CORR_Ubuntu
- NTV_CORR_Windows
to compare not only status execution but also the output results.
For tests:
- tests/acceptance_tests/test_*.py
- tests/integration_tests/redis/test_*.py
- tests/integration_tests/adapters/binance/test_binance_execution.py
- tests/integration_tests/adapters/ccxt/test_ccxt_providers.py
- tests/integration_tests/adapters/oanda/test_oanda_factory.py
- tests/integration_tests/adapters/oanda/test_oanda_providers.py
- tests/performance_tests/test_*.py
- tests/unit_tests/analysis/test_*.py
- tests/unit_tests/backtest/test_*.py
- tests/unit_tests/common/test_*.py
- tests/unit_tests/core/test_*.py
- tests/unit_tests/data/test_*.py
- tests/unit_tests/execution/test_*.py
- tests/unit_tests/indicators/test_*.py
- tests/unit_tests/live/test_*.py
- tests/unit_tests/model/test_*.py
- tests/unit_tests/risk/test_*.py
- tests/unit_tests/serialization/test_*.py
- tests/unit_tests/trading/test_*.py
NTV_ORIG_Ubuntu, NTV_CORR_Ubuntu and NTV_CORR_Windows give the same outputs and
status 'ok'
Notes:
- for tests/unit_tests/backtest/test_backtest_models.py test
the results are different with status 'failed'.
The reason - pseudorandom number generators ...
- for tests/performance_tests/test_*.py tests
1) performance results for NTV_CORR_Ubuntu and NTV_ORIG_Ubuntu are
practically identical (plus-minus)
2) performance results for NTV_CORR_Ubuntu and NTV_ORIG_Ubuntu are faster
by 10-100% then performance results for NTV_CORR_Windows !!!
For tests:
- tests/integration_tests/live/test_live_node.py
ImportError: ccxtpro is not installed (*)
- tests/integration_tests/adapters/binance/test_binance_factory.py
ImportError: ccxtpro is not installed (*)
- tests/integration_tests/adapters/bitmex/test_bitmex_execution.py
Status ok
Outputs for NTV_CORR_Ubuntu, NTV_ORIG_Ubuntu and NTV_CORR_Windows are different
- tests/integration_tests/adapters/bitmex/test_bitmex_factory.py
ImportError: ccxtpro is not installed (*)
- tests/integration_tests/adapters/ccxt/test_ccxt_data.py
Status ok
But for NTV_CORR_Ubuntu, NTV_ORIG_Ubuntu (not for NTV_CORR_Windows)
after execution all tests the next message is outputed:
"Fatal Python error: could not acquire lock for <_io.BufferedWriter name='<stdout>'>
at interpreter shutdown, possibly due to daemon threads"
....
- tests/integration_tests/adapters/ccxt/test_ccxt_execution.py
Status ok
Need your explanations
- tests/integration_tests/adapters/ccxt/test_ccxt_factory.py
ImportError: ccxtpro is not installed (*)
- tests/integration_tests/adapters/ib/test_ib_providers.py
ImportError: ccxtpro is not installed (*)
- tests/integration_tests/adapters/oanda/test_oanda_data.py
Status ok
Need your explanations
(*) - No subscription to ccxtpro
Notes:
I think it's clear these results has no relationship to proposed changes.
At these moment I don't see serious technical problems to support Windows.
There is only 3 Low-level code compatibility problems.
Approaches fixing the build errors on Windows see above.
Also at these moment I don't see serious maintenence problems to support Windows (???).
You can choose more flexible policy:
make proposed changes without declaring Windows support officially or when you wiil be sure
I can send you NTV_CORR version for control (say how?)
Thanks for your work !
I'm looking at the ccxt.pro source code, but I'm not 100% sure if live trading is going well.
It is also a bit uneasy that some exchanges' data checksum logics are left as TODO.
Does it work well?
So I'm just quickly documenting some of my performance profiling, following on from conversations here #190
So if the event loop is warm it takes ~35μs to generate a market order, pass through the queue in the execution engine and get it to the execution client. If epoll is active and it needs to be woken up it takes 10x that amount. Code can be found in performance_tests/test_perf_live_execution.py
That's a pretty good start. We can't compete in the ULL nanosecond space but that was never the intention for this platform.
A couple of things I can see immediately need improving are UUID and timestamp generation.
Unfortunately the UUID I refactored and c typed from the CPython source isn't much faster than the built-in (see those perf tests), although its still better to use because it can be c typed. There's a library out there fastuuid
which provides Python bindings for the Rust UUID however out of the box that cant be c typed.
For the timestamp there are calls to clock.utc_now()
which under the hood makes a pure Python call datetime.now(tz=pytz.utc)
. I've been trying to figure out how to get a POSIX directly off the system clock, there are some includes from Cython but haven't figured it out as yet.
Currently the Order is identified through the id
member. However, when tracking the order in real live executions, the brokers usually let's the client identify the order through a client_id
which is a unique id set by the client, as opposed to the exchange_id
which is set upon accepting the order by the exchange. I recommend implementing client_id
the Order, and be passed as optional when creating an order.
The order flow I envision is as follows for tracking an order:
Open for comments;
My thoughts here are that these days not every tradable asset is necessarily a Security in the traditional sense.
So this name could be more general?
Scaffolding for an Interactive Brokers integration leveraging the ib_insync
library has been added.
The vision is to initially support futures trading.
Whilst many quant traders are using bespoke machine learning models / proprietary indicators, it would still be valuable to add some more "standard" indicators for the more traditional algo traders.
Putting out an open invitation for anyone to add an indicator including unit tests.
Alternatively I'm "taking requests" and suggestions for indicators!
Right now the commission is calculated only when the position is reduced. I think this is wrong, because the commission should be applied also when a fill happens in the same dir as the position. I can also submit a fix to this if you agree.
On a separate note, it would be nice to have different commissions applied for maker/taker, where in case of maker commission bp can be negative (rebates).
Currently commission and funding rates are expressed in basis points. For simplicity and standardization I'm proposing this be changed to a decimal percentage e.g.
1% = Decimal("1")
-0.0250% = Decimal("-0.0250")
This seems to align with most fee schedules.
Thoughts/comments?
I'm documenting the decision to remove Rust from the nautilus_trader
codebase.
After spending weeks attempting to come up with the optimal integration solution I've decided that the increase in complexity with including another language in the tool/build chain is not worth it.
Getting Rust objects to be C ABI and FFI compatible was involving alot of raw pointer mangling which in turn required large blocks of code marked unsafe
in Rust-land. Because of this one of the main value propositions of Rust (memory-safety) was being compromised, with a rising potential for memory leaks as data was being exchanged across the Python <-> Rust boundary.
I still believe there is huge potential in the Rust language, and will continue to work with it. However not at this stage attempting to splice it into the Python runtime. I think its far safer and simpler to utilize the language with other services across a serialized message boundary, with a greenfield codebase written entirely in Rust and no unsafe external C calls. For example to build a data aggregator, smart-order-router or centralized execution engine which can handle messages from many nautilus_trader boxes.
For the performance goals of the platform - Cython is a more than adequate tool, allowing direct access to the underlying CPython API, with plenty of potential still to be unlocked.
Currently the Money
object hard codes a precision of 2 ala Fiat money.
Since cryptographic money can be denominated to much a higher precision, this will need to be accommodated. This means there's potential for Currency to become a first class object rather than an enum, carrying with it a precision and maybe even a CurrencyType
enum FIAT
, CRYPTO
etc if this may affect logic elsewhere.
Opened this to flag the issue and open to suggestions...
Placing a Limit Order below the market price, should execute the order at the market price. In this case the order fill structure should capture this as fill_event.order_type
= OrderType.Limit
and fill_event.execution.order_type
= OrderType.Market
.
A quick solution would be to convert the order type to Market
when placing the Limit
below/equal the market price, and let it be executed as MARKET. This doesn't imply changing the OrderFillEvent
.
Please let me know your thoughts.
Builds for windows-latest
(Windows Server 2019) are failing for very low level reasons e.g.
build/optimized\nautilus_trader\core\message.c(1402): error C2061: syntax error: identifier '__uint128_t'
Which is a c typedef for a 128 bit unsigned integer in the UUID type.
See more details here;
a9e99df
I'm fielding these questions to the community
Also, if anyone with knowledge has ideas on how to approach fixing the build errors - I'm open to suggestions.
Redis support on Windows is also a potential issue.
Maybe it already works?
Adding this as an issue to remind us to enable it in CI and see how it goes.
I've been struggling with how to get pre-computed data (e.g. some other data source, expensive to compute indicator, etc.) fed into the strategy alongside the bar or tick data during a backtest and looking for any advice. When all prices are known ahead of time (i.e. backtest) it's often preferable to compute these indicators ahead of time. Or, maybe you have a news data feed that serves data at timestamps that don't necessarily coincide with tick or bar timestamps.
The library is not currently set up to do this and feel it may be beneficial to others.
Ideas I've tried that both work but both seem hacky and don't fit the clean organization of the project:
Create an additional data dictionary in the BacktestDataClient alongside the existing _symbols, _price_volume, and _timestamps called _aux_features. During the initialization phase where data is added (e.g data.add_bars(), data.add_ticks(), etc.) an additional method is added called *.add_aux_features() to add any pre-computed data. Changes to various other classes must be made to keep these aux_features synced with the correct OHLCV timestamp.
Add these pre-computed features (called aux_features) to the DataCache when BacktestDataClient.setup() is called. The advantage here is that BacktestDataClient.setup() slices the full dataset between start:stop dates so the aux_features can be sliced between these same start:stop dates. The disadvantage is it's using the cache improperly as the normal OHLCV data is not added to the cache until after it is processed (i.e. no future prices exist in the cache). This works because the strategy has access to the cache and this provides a method to access pre-computed data by index inside a strategy at customizable intervals such as on_bar() or on_tick(). The index must be known so we must track the number of bars or ticks that have been seen by the strategy to retrieve the correct index.
An indicator that handles this type of data and correctly feeds it according to the current timestamps seems best but indicators do not have access to any future data.
Can you think of a better way to implement this? I can continue on my own if this doesn't fit the broader scope of this awesome project.
Open for comments
There has been some efforts to add support for Quanto instruments (where settlement currency is different to base or quote).
It's been decided for now that a Position
is only responsible for tracking it's PNL in either base or quote currency (depending on if it's inverse or not).
Hooks remain on Instrument
to work out Quanto settlement with an exchange rate to settlement currency. Currently instruments which specify a third settlement currency in the constructor have the settlement currency set to either base or quote depending on if inverse.
This decision was made so as not to introduce the foreign concept of say a "cost currency" and so that code can keep referring to settlement_currency
. So to start reviving this capability we just need to change the Instrument
constructor to set settlement_currency
as such, and ensure the xrate is passed into the appropriate methods.
Also, we can revive the CostSpecification
classes out of source control if we want to recommission that concept 2be6ca7.
We need to refactor the BacktestDataContainer.add_bar_data
method so that data added with PriceType.LAST
will be interpreted as, and pre-processed into, trade ticks.
I'd like to open a discussion about things that could be improved on the backtest / simulation side. A handful of things I'd like to see in the near term:
I think it would be pretty useful to incorporate commission in the order fills and position. For position the commission should be part of the realised PNL as well.
If you agree on this, I can open a PR to support the above.
Lets push ahead with setting up some scaffolding for a CCXT integration.
I'm proposing we keep all broker and exchange integration modules in nautilus_trader/adapters
. I'm adding a ccxt
module folder on the next push.
Basically what needs to happen is implementations of DataClient
and ExecClient
need to be written.
The data classes are about to get refactored to ease this whole process though, I'm thinking of separating DataClient
from a DataManager
type class (for lack of a better name). Then we can have as many DataClients
as we like all piping back to DataManager
.
Thoughts everyone?
An early version of the OrderBook
feature has been pushed to the develop
branch.
The bids and asks in the order book are represented as lists of (Price, Quantity)
tuples, in each case sorted from top to bottom. I feel this is a more usable representation than lists of [float, float]
which is what's currently returned from CCXT? There is some overhead (~0.5μs / 500ns) to construct each object - which would probably be OK for most users?
So a user can subscribe to an order book by symbol. OrderBook
object snapshots will then be passed into on_order_book
from the DataEngine
. Optionally a user can specify a timedelta
interval if they only require periodic snapshots (also with an optional start_delay timedelta
if they don't want those snapshots to occur at floored times i.e the very start of a minute or second).
Some things to consider is how to specify the level and depth required?
Separate levels would then require separate socket streams for L2 and L3. I'm currently thinking depth will be the maximum any strategy has currently subscribed to for that symbol.
Requesting any comments on the feature.
I'd like to recommand default store of tick data as a parquet not csv.
In a nutshell, parquet is a highly well-designed file type for handling time series data-sets such as large tick data. Pandas supports very well operation without any boilerplate code like pd.read_csv().
The main reason I use this is because it can do compression. You can compress a bitcoin trade tick for a day to around 30mb (200mb when uncompressed, You can learn more by looking at the sample data posted on PR.)
I used parquet mainly by previous HFT firm and it worked fine without any problems. so I also recommanded to use this in the nautilus_trader project.
Pay attention to the possibility to detect bugs/improvements/corrections using Cython compiler directives
# build.py
CYTHON_COMPILER_DIRECTIVES = {
...
# Warns about use of variables that are conditionally uninitialized.
# (default: False)
# "warn.maybe_uninitialized": True - allows to detect many bugs and corrections.
"warn.maybe_uninitialized": True,
# Warns about unused variables and declarations.
# (default: False)
# "warn.unused": True - allows to detect many bugs and corrections.
"warn.unused": True,
# Warns about unused function arguments.
# (default: False)
# "warn.unused_arg": True - allows to detect many bugs and corrections.
"warn.unused_arg": True,
# Warns about unused assignment to the same name.
# (default: False)
# "warn.unused_result": True - allows to detect many bugs and corrections.
"warn.unused_result": True,
}
Only some examples:
1) "warn.maybe_uninitialized" directive
Setting the directive to 'True' allows to identify next bugs:
file: \adapters\ccxt\execution.pyx
method: CCXTExecutionClient.generate_order_status_report
bug: variable 'state' is uninitialized for all possible cases,
append 'raise' for unsupported 'status' types
file: \backtest\data_producer.pyx
method: CachedProducer.next
bug: variable 'data' is uninitialized for 'None' if there are no data
file: \backtest\engine.pyx
method: BacktestEngine.__init__
bug: variable 'data_client' is not defined for 'all' client_type ...,
append 'raise' for unsupported 'client_type' types
file: \backtest\exchange.pyx
method: SimulatedExchange.process_order_book
bug: variables 'bid' and 'ask' are not defined for all possible cases,
append 'raise' for unsupported 'OrderBookData' types
...
NEED the detail analysis for each warning !!!
2) "warn.unused" directive
Setting the directive to 'True' allows to identify next 'bugs':
file: \backtest\exchange.pyx
method: SimulatedExchange._build_current_bid_rates
warning: "Unused entry 'instrument_id'"
improvment:
# current version:
cdef inline dict _build_current_bid_rates(self):
cdef InstrumentId instrument_id
cdef QuoteTick tick
return {instrument_id.symbol.value: price.as_decimal() for instrument_id, price in self._market_bids.items()}
# new version 1:
# declaring explicitly Cython's variables has no effect in this case
cdef inline dict _build_current_bid_rates(self):
return {instrument_id.symbol.value: price.as_decimal() for instrument_id, price in self._market_bids.items()}
# new version 2:
# Use explicitly Cython's variables and 'for'
# Can really speed up code execution to build rates for xrate_calculation!!!
cdef inline dict _build_current_bid_rates(self):
cdef InstrumentId instrument_id
cdef Price price
cdef dict rates = dict()
for instrument_id, price in self._market_bids.items():
rates[instrument_id.symbol.value] = price.as_decimal()
return rates
file: \data\engine.pyx
method: DataEngine._handle_bars
warning: "Unused entry 'TimeBarAggregator'"
improvment: declare the target variable
# current version:
....
cdef TimeBarAggregator
....
# new version:
....
cdef TimeBarAggregator aggregator
....
...
NEED the detail analysis for each warning !!!
3) "warn.unused_arg" directive
Setting the directive to 'True' allows to identify next improvements:
file: \indicators\atr.pyx
method: AverageTrueRange.__init__
warning: "Unused argument 'check_inputs'"
improvment: remove arg 'check_inputs'
file: \model.instrument.pyx
method: Instrument.__init__
warning: "Unused argument 'financing'"
improvment: remove arg 'financing' or define the correspondent data member ...
... (there are tons ща false warnings)
NEED the detail analysis for each warning !!!
4) "warn.unused_result" directive
Setting the directive to 'True' allows to identify next improvements:
file: \execution\engine.pyx
method: ExecutionEngine.__init__
warning: "Unused result in 'config'"
improvment: remove arg 'config' or define the correspondent data member ...
file: \risk\engine.pyx
method: RiskEngine.__init__
warning: "Unused result in 'config'"
improvment: remove arg 'config' or define the correspondent data member ...
...
NEED the detail analysis for each warning !!!
Thanks
model\
bar.pxd,bar.pyx,tick.pxd,tick.pyx,
data\
aggregation.pxd,aggregation.pyx ...
(see also (3))
a) remove the member 'bar_spec' from BarBuilder class
cdef readonly BarSpecification bar_spec
'bar_spec' is defined in BarBuilder constructor as
self.bar_spec = bar_type.spec
There are only 2 cases to do this:
1) self.bar_spec can be changed later (independently from value self.bar_type.spec)
2) to speed up code execution (significant)
This is not our case.
similar cases:
b) remove the members 'symbol', 'venue' from BarType class
cdef readonly Symbol symbol
cdef readonly Venue venue
c) remove the members 'symbol', 'venue' from Tick class
cdef readonly Symbol symbol
cdef readonly Venue venue
d) remove the member 'step' from TickBarAggregator class (see TimeBarAggregator)
cdef readonly int step
e) remove the member 'step' from VolumeBarAggregator class (see TimeBarAggregator)
cdef readonly int step
f) remove the member 'step' from ValueBarAggregator class (see TimeBarAggregator)
cdef readonly int step
g) ...
model\bar.pxd, bar.pyx (see also (3))
rename the member 'type' to 'bar_type' in Bar class
In BarBuilder class the correspondent member is ...
cdef readonly BarType bar_type
model\bar.pxd,bar.pyx, data\aggregation.pxd,aggregation.pyx ...
More serious question, before it's too late (!!!).
Redesign BarType, BarBuilder ... to exclude InstrumentId instrument_id
(try to keep only BarSpecification)
For BarAggregator InstrumentId must be passed explicitly to the constructor
In bar.pyx instrument_id is only used in
- __str__ and __hash__ methods
(instrument_id is not essential info, more important - spec )
- serializing methods (really no need for a single bar)
In aggregation.pyx instrument_id (as part of bar_type) is only used:
- for creating named timer
(we can build name for timer from pair instrument_id and spec)
- on call _handler and callback
(we can pass an instrument_id as arg)
The main idea - don't produce redundant/confusing/... info.
core\uuid.pyx
a) Remove
assert 0 <= self.int_val < 1 << 128, "int is out of range (need a 128-bit value)"
This statement has no effect.
self.int_val was created from 16 bytes with default value of sign=False
self.int_val = int.from_bytes(value, byteorder="big")
Any 16 bytes will be converted to the int value in the range [0, 1 << 128)
b) Use self.int_value (not self.value) in comparison operations __eq__ and __ne__.
It's more logical if we use self.int_value in __lt__, __gt__, ... operations
to be continued
incorrectly calculated - always equal to the initial value of the account, regardless of positions and deals
class Testing(TradingStrategy):
....
self.free.append(self.execution.accounts()[0].balance_free(USDT))
self.locked.append(self.execution.accounts()[0].balance_locked(USDT))
self.balance.append(self.execution.accounts()[0].balance(USDT))
The Oanda adapter is currently a work in progress.
Oanda is a brokerage popular with retail traders https://www.oanda.com. It was chosen as an integration mainly due to the quality of their API, and range of trading products offered.
The OandaDataClient
has some functionality however the OandaExecutionClient
hasn't been started as yet.
As far as I understand, the calculation of FX broker commissions differs from cryptocurrency exchanges - they are considered debited from the balance immediately after the order is executed, respectively, for a backtest close to reality, it would be nice to take this into account, or at least have access to exchange.total_commissions.values () from Strategy so that in manually subtract them from the current equity.
021-03-18T16:53:27.626636Z [INF] BacktestEngine: =================================================================
2021-03-18T16:53:27.626668Z [INF] BacktestEngine: BINANCE-001
2021-03-18T16:53:27.626717Z [INF] BacktestEngine: =================================================================
2021-03-18T16:53:27.626762Z [INF] BacktestEngine: Account balances (starting): 100,000.00000000 USDT
2021-03-18T16:53:27.626783Z [INF] BacktestEngine: Account balances (ending): 113,776.56650000 USDT
2021-03-18T16:53:27.626800Z [INF] BacktestEngine: Commissions (total): 17,808.93350000 USDT
2021-03-18T16:53:27.626816Z [INF] BacktestEngine: =================================================================
I'm not sure if registering an indicator for quotes should be supported or not, but currently it raises an error. If not supported, would be nice to have a condition in place that restricts registering the indicators for quotes.
# Register the indicators for updating
self.register_indicator_for_quote_ticks(
symbol=self.symbol,
indicator=self.sma)
Traceback (most recent call last):
File "/Users/solo/git/nautilus_trader/backtest/backtest_market_maker.py", line 74, in <module>
engine.run(start, stop)
File "nautilus_trader/backtest/engine.pyx", line 195, in nautilus_trader.backtest.engine.BacktestEngine.run
File "nautilus_trader/backtest/engine.pyx", line 282, in nautilus_trader.backtest.engine.BacktestEngine.run
File "nautilus_trader/backtest/data.pyx", line 368, in nautilus_trader.backtest.data.BacktestDataClient.process_tick
File "nautilus_trader/common/data.pyx", line 221, in nautilus_trader.common.data.DataClient.handle_quote_tick
File "nautilus_trader/trading/strategy.pyx", line 463, in nautilus_trader.trading.strategy.TradingStrategy.handle_quote_tick
File "nautilus_trader/trading/strategy.pyx", line 478, in nautilus_trader.trading.strategy.TradingStrategy.handle_quote_tick
File "nautilus_trader/indicators/base/indicator.pyx", line 51, in nautilus_trader.indicators.base.indicator.Indicator.handle_quote_tick
TypeError: 'NotImplementedType' object is not callable
I believe the first line of each code snippet contains a typo. Unless I'm misunderstanding all prices in lines #172 and #176 should be df_ticks_o
.
nautilus_trader/nautilus_trader/data/wrangling.pyx
Lines 172 to 175 in 0c410dd
nautilus_trader/nautilus_trader/data/wrangling.pyx
Lines 176 to 179 in 0c410dd
2021-01-10T19:53:31.983Z [INF] BacktestEngine: =================================================================
2021-01-10T19:53:31.983Z [INF] BacktestEngine: NAUTILUS TRADER - Algorithmic Trading Platform
2021-01-10T19:53:31.983Z [INF] BacktestEngine: by Nautech Systems Pty Ltd.
2021-01-10T19:53:31.983Z [INF] BacktestEngine: Copyright (C) 2015-2020. All rights reserved.
2021-01-10T19:53:31.983Z [INF] BacktestEngine: =================================================================
2021-01-10T19:53:31.983Z [INF] BacktestEngine:
2021-01-10T19:53:31.983Z [INF] BacktestEngine: .......
2021-01-10T19:53:31.983Z [INF] BacktestEngine: .............
2021-01-10T19:53:31.983Z [INF] BacktestEngine: . ......... .......
2021-01-10T19:53:31.984Z [INF] BacktestEngine: . ......... .. .......
2021-01-10T19:53:31.984Z [INF] BacktestEngine: . ......',,,,'..........
2021-01-10T19:53:31.984Z [INF] BacktestEngine: .. ......::,,''';,.........
2021-01-10T19:53:31.984Z [INF] BacktestEngine: .. ....'o:;oo;..:'..... ''
2021-01-10T19:53:31.984Z [INF] BacktestEngine: .. ......,;,,..,:'.........
2021-01-10T19:53:31.984Z [INF] BacktestEngine: .. .........';:'..... ...
2021-01-10T19:53:31.984Z [INF] BacktestEngine: .. .......'..... .'. .'
2021-01-10T19:53:31.984Z [INF] BacktestEngine: .. ..... .. .. ..
2021-01-10T19:53:31.984Z [INF] BacktestEngine: .. .' ....
2021-01-10T19:53:31.984Z [INF] BacktestEngine: .. .. .'.
2021-01-10T19:53:31.984Z [INF] BacktestEngine: .... .....
2021-01-10T19:53:31.984Z [INF] BacktestEngine: .... ..'..
2021-01-10T19:53:31.984Z [INF] BacktestEngine: ..................
2021-01-10T19:53:31.984Z [INF] BacktestEngine:
2021-01-10T19:53:31.984Z [INF] BacktestEngine: =================================================================
2021-01-10T19:53:31.984Z [INF] BacktestEngine: SYSTEM SPECIFICATION
2021-01-10T19:53:31.984Z [INF] BacktestEngine: =================================================================
2021-01-10T19:53:31.984Z [INF] BacktestEngine: CPU architecture: x86_64
2021-01-10T19:53:31.985Z [INF] BacktestEngine: CPU(s): 8 None
2021-01-10T19:53:31.986Z [INF] BacktestEngine: RAM-Total: 17,093 MB
2021-01-10T19:53:31.986Z [INF] BacktestEngine: RAM-Used: 14,426 MB (84.6%)
2021-01-10T19:53:31.986Z [INF] BacktestEngine: RAM-Avail: 2,625 MB (15.4%)
2021-01-10T19:53:32.011Z [INF] BacktestEngine: OS: Linux-4.4.0-18362-Microsoft-x86_64-with-glibc2.29
2021-01-10T19:53:32.011Z [INF] BacktestEngine: =================================================================
2021-01-10T19:53:32.011Z [INF] BacktestEngine: VERSIONING
2021-01-10T19:53:32.011Z [INF] BacktestEngine: =================================================================
2021-01-10T19:53:32.011Z [INF] BacktestEngine: nautilus-trader 1.94.6
2021-01-10T19:53:32.011Z [INF] BacktestEngine: python 3.8.5
2021-01-10T19:53:32.011Z [INF] BacktestEngine: numpy 1.19.5
2021-01-10T19:53:32.011Z [INF] BacktestEngine: scipy 1.6.0
2021-01-10T19:53:32.011Z [INF] BacktestEngine: pandas 1.2.0
2021-01-10T19:53:32.011Z [INF] BacktestEngine: =================================================================
2021-01-10T19:53:32.011Z [INF] BacktestEngine: Building engine...
2021-01-10T19:53:32.011Z [INF] BypassExecutionDatabase: Initialized.
2021-01-10T19:53:32.011Z [INF] DataEngine: state=INITIALIZED...
2021-01-10T19:53:32.011Z [INF] DataCache: Initialized.
2021-01-10T19:53:32.011Z [INF] DataEngine: use_previous_close=False
2021-01-10T19:53:32.011Z [INF] BacktestDataProducer: Preparing BTC/USDT.BINANCE data...
2021-01-10T19:53:32.011Z [INF] BacktestDataProducer: Prepared 2,325,039 BTC/USDT.BINANCE quote tick rows in 7.63s.
2021-01-10T19:53:32.011Z [INF] BacktestDataProducer: Prepared 7,803,194 BTC/USDT.BINANCE trade tick rows in 17.26s.
2021-01-10T19:53:32.011Z [INF] BacktestDataProducer: Merging tick data streams...
Traceback (most recent call last):
File "./examples/backtest/backtest_btcusdt_trade_ticks.py", line 65, in <module>
engine = BacktestEngine(
File "nautilus_trader/backtest/engine.pyx", line 220, in nautilus_trader.backtest.engine.BacktestEngine.__init__
self._data_producer = BacktestDataProducer(
File "nautilus_trader/backtest/data_producer.pyx", line 170, in nautilus_trader.backtest.data_producer.BacktestDataProducer.__init__
self.min_timestamp = max(self._quote_tick_data.index, self._trade_tick_data.index)
File "/home/deploy/.cache/pypoetry/virtualenvs/nautilus-trader-keG6LY4b-py3.8/lib/python3.8/site-packages/pandas/core/indexes/extension.py", line 134, in wrapper
return op(other)
File "/home/deploy/.cache/pypoetry/virtualenvs/nautilus-trader-keG6LY4b-py3.8/lib/python3.8/site-packages/pandas/core/ops/common.py", line 65, in new_method
return method(self, other)
File "/home/deploy/.cache/pypoetry/virtualenvs/nautilus-trader-keG6LY4b-py3.8/lib/python3.8/site-packages/pandas/core/arraylike.py", line 45, in __gt__
return self._cmp_method(other, operator.gt)
File "/home/deploy/.cache/pypoetry/virtualenvs/nautilus-trader-keG6LY4b-py3.8/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py", line 932, in _cmp_method
other = self._validate_comparison_value(other)
File "/home/deploy/.cache/pypoetry/virtualenvs/nautilus-trader-keG6LY4b-py3.8/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py", line 454, in _validate_comparison_value
raise ValueError("Lengths must match")
ValueError: Lengths must match
INPUT DATA: Quotes(7,803,194 rows), Trade(2,325,039 row)
I don't think it should be like this, but there is error.
I'm fixing this problem right now, but I'd like to ask a question to be sure. Is it correct that the backtest system can take two streams of different lengths as inputs?
I wonder if nautilus_trader could support the strategy which has multiple timeframes (eg. both daily and minute data)?
The network sub-package will contain generic code for performant WebSocket and HTTP clients written in Cython.
The base classes can be inherited to assist with writing adapters for integrations with exchanges, brokerages, data providers/feeds and other venues or services.
Open for comments
Proposed changes as per commit 92d7a33
Added more decimal.Decimal
type checks in commit 3354c03
There has been an ongoing challenge to create the best API possible for the domain value types in the framework, namely Price
, Quantity
and Money
, which all inherit from a common base class wrapping a built-in decimal.Decimal
.
There have been several iterations of this API, each improving on the last, often with feedback and advice from core contributors.
The goal is to provide;
float
types and decimalsPreviously the common base class was named Decimal
and was also used as a first class object throughout the framework, often being used to represent anything with a fixed precision which wasn't a defined domain value type (as above).
Considering this for some time, I came to the conclusion that there was a mixing of concerns and concepts. The approach of using the base class to also represent other fixed precision concepts/types exposed it in a way that a user may find confusing, especially that it shared a name with the built-in decimal.Decimal
.
To clarify this the name has been changed to BaseDecimal
, with its use only intended as the parent class for the domain value types. The type in general provides in my opinion a better experience handling precision and rounding, and allows float
to be used as operands in math ops.
If a user includes a float
in a math op then the type of the returned result is a float
. The thinking behind this is that a float
has no precision specified and conversion to a decimal possibly implies a precision the user did not intend.
This has allowed many redundant and uncesseary instantiations of BaseDecimal
within framework methods and functions to be removed, with operations mainly on the decimal.Decimal
type.
Following on from this all uses of BaseDecimal
(where it wasn't being used as a base class as above, or in an operation with one of its subclasses) have been changed to decimal.Decimal
.
The only down side I can see from this is that some typing information has been lost, as Cython cannot recognize decimal.Decimal
as a C object, just a general PyObject
.
To ensure static type safety remains some efficient Condition.type(value, decimal.Decimal, "value")
checks have been added.
The returned type from math ops with a type other than float
has been changed to a vanilla decimal.Decimal
(changed from the BaseDecimal
).
Performance has been ever so slightly improved on the backtest performance benchmark ~5.809 seconds vs ~5.960 seconds, although this was not the primary goal.
Does returning a float
from a math op involving a float
seem like expected and reasonable behavior?
Does returning a decimal.Decimal
from all other valid math ops seem like expected and reasonable behavior?
Should comparisons between float
and BaseDecimal
be considered invalid (raise an exception)? This is the currency behavour. Or should it be as per the built-in decimal.Decimal
which allows it.
Does raising a TypeError
when the BaseDecimal
constructor is passed a float
value with no precision passed seem like expected and reasonable behavior?
Are there any other areas for improvement regarding this API, or is there any feedback from users?
A RiskEngine
has begun its implementation journey.
Currently the engine is wired into the platform after the ExecutionEngine
and is an optional component (can be bypassed).
The current vision for capabilities include;
After some research it seems like a good idea if we transition the performance benchmarking over to a more established tool such as pytest-benchmark
https://pytest-benchmark.readthedocs.io/en/latest/.
This will allow continuous benchmarking through GitHub workflows which publishes on a page to track and analyze the benchmarks https://github.com/marketplace/actions/continuous-benchmark.
This will then allow the removal of the bespoke PerformanceHarness
which is really reinventing the wheel which pytest is already rolling with.
Development on this branch:
https://github.com/nautechsystems/nautilus_trader/tree/performance
Thoughts and comments welcome!
Since now we can differentiate between the maker and taker trades based on the EventFill.liquidity_side
, it would be appropriate to apply the commission WRT the liquidity side.
Shall I open a PR for this?
There is an ongoing effort to convert all docstrings to the numpydoc standard.
https://numpydoc.readthedocs.io/en/latest/format.html
Once this is complete then a readthedocs
website can be established.
https://readthedocs.org/nautilus_trader
Currently the position is represented in quote currency. I find this confusing, hence I propose to change it to base currency.
The reset() method in BacktestDataProducer hints that running a backtest with the same data can be repeated
I have not found any trace of its use on the system, but the idea is good
maybe it makes sense to cache the results of the first launch so as not to create QuoteTick, TradeTick every next time
the difference is approximately as follows:
~2M trades & quotes just iteration without time for preparation:
7.48 sec VS 0.67 sec (if cached and stored in SortedDict)
How valuable is it?
Early beta versions of live integrations for Binance
(spot) and BitMEX
are now available for testing.
Examples of how to set these up can be found in the examples
folder.
I'd suggest using a separate account with a very small balance at this stage.
One caveat is that currently the ccxtpro
package needs to be installed from their private repo (which in turn requires a license).
# if you're using Git/HTTPS authentication
pip3 install git+https://github.com/kroitor/ccxt.pro.git#subdirectory=python
# if you are connecting to GitHub with SSH
pip3 install git+ssh://[email protected]/kroitor/ccxt.pro.git#subdirectory=python
The ACCOUNT_ID
environment variable key isn't required. It's available if a user wishes to identify an account in some way which could become useful for multiple/sub accounts on the same exchange. My own use case has been to assign the actual account identifier which matches the API_KEY
and API_SECRET
.
Requesting comments from users willing to test this capability
pyarrow
now has working wheels for Python 3.9 and those tests are passing.
So I've removed all the pytest skips and returned the codecov
workflow to Python 3.9
**engine.add_exchange(
venue=BINANCE,
oms_type=OMSType.NETTING,
generate_position_ids=False,
starting_balances=[Money(1_000, USDT), Money(1, BTC)],
fill_model=fill_model,
)**
makes it possible to buy for 424580 USDT
bug or is it intended?
in live mode did not check...
Loading instruments...
2021-03-18T14:10:52.607303Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:52.607336Z [INF] BacktestEngine: Building engine...
2021-03-18T14:10:52.607359Z [INF] BypassExecutionDatabase: Initialized.
2021-03-18T14:10:53.462142Z [INF] BacktestEngine: Initialized in 0.855s.
2021-03-18T14:10:53.462175Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.462186Z [INF] BacktestEngine: MEMORY USAGE
2021-03-18T14:10:53.462194Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.462265Z [INF] BacktestEngine: RAM-Total: 68,719 MB
2021-03-18T14:10:53.462278Z [INF] BacktestEngine: RAM-Used: 35,052 MB (58.00%)
2021-03-18T14:10:53.462288Z [INF] BacktestEngine: RAM-Avail: 28,835 MB (42.00%)
2021-03-18T14:10:53.682557Z [INF] BacktestEngine: Data size: 42.65 MB
2021-03-18T14:10:53.682868Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.682885Z [INF] BacktestEngine: BACKTEST RUN
2021-03-18T14:10:53.682895Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.682908Z [INF] BacktestEngine: Run started: 2021-03-18T14:10:53.682Z
2021-03-18T14:10:53.682924Z [INF] BacktestEngine: Backtest start: 2020-08-14T10:00:00.223Z
2021-03-18T14:10:53.682935Z [INF] BacktestEngine: Backtest stop: 2020-08-14T14:59:58.693Z
2021-03-18T14:10:53.682944Z [INF] BacktestEngine: Execution resolution: ETH/USDT.BINANCE=TICK
2021-03-18T14:10:53.682951Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.682961Z [INF] BacktestEngine: BINANCE-001
2021-03-18T14:10:53.682971Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.682983Z [INF] BacktestEngine: Account balances (starting): 1,000.00000000 USDT, 1.00000000 BTC
2021-03-18T14:10:53.682991Z [INF] BacktestEngine: Setting up backtest...
2021-03-18T14:10:53.683423Z [INF] BacktestEngine: Reset.
2021-03-18T14:10:53.683475Z [INF] BacktestEngine: Running backtest...
2021-03-18T14:10:53.843135Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.843162Z [INF] BacktestEngine: BACKTEST DIAGNOSTICS
2021-03-18T14:10:53.843173Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.843185Z [INF] BacktestEngine: Run started: 2021-03-18T14:10:53.682Z
2021-03-18T14:10:53.843196Z [INF] BacktestEngine: Run finished: 2021-03-18T14:10:53.843Z
2021-03-18T14:10:53.843208Z [INF] BacktestEngine: Backtest start: 2020-08-14T10:00:00.223Z
2021-03-18T14:10:53.843218Z [INF] BacktestEngine: Backtest stop: 2020-08-14T14:59:58.693Z
2021-03-18T14:10:53.843228Z [INF] BacktestEngine: Elapsed time: 0:00:00.160263
2021-03-18T14:10:53.843237Z [INF] BacktestEngine: Execution resolution: ETH/USDT.BINANCE=TICK
2021-03-18T14:10:53.843246Z [INF] BacktestEngine: Iterations: 69,806
2021-03-18T14:10:53.843254Z [INF] BacktestEngine: Total events: 151
2021-03-18T14:10:53.843266Z [INF] BacktestEngine: Total orders: 30
2021-03-18T14:10:53.843277Z [INF] BacktestEngine: Total positions: 15
2021-03-18T14:10:53.843287Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.843297Z [INF] BacktestEngine: BINANCE-001
2021-03-18T14:10:53.843305Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.843322Z [INF] BacktestEngine: Account balances (starting): 1,000.00000000 USDT, 1.00000000 BTC
2021-03-18T14:10:53.843331Z [INF] BacktestEngine: Account balances (ending): -9,150.00000000 USDT, 1.00000000 BTC
2021-03-18T14:10:53.843339Z [INF] BacktestEngine: Commissions (total): 12,756.89000000 USDT
2021-03-18T14:10:53.843349Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.843358Z [INF] BacktestEngine: PERFORMANCE STATISTICS
2021-03-18T14:10:53.843365Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.854684Z [INF] BacktestEngine: USDT
2021-03-18T14:10:53.854711Z [INF] BacktestEngine: -----------------------------------------------------------------
2021-03-18T14:10:53.854826Z [INF] BacktestEngine: PnL: -10,150.0 USDT
2021-03-18T14:10:53.854840Z [INF] BacktestEngine: PnL%: -1015.0%
2021-03-18T14:10:53.854848Z [INF] BacktestEngine: Max Winner: 1,638.73 USDT
2021-03-18T14:10:53.854856Z [INF] BacktestEngine: Avg Winner: 1,638.73 USDT
2021-03-18T14:10:53.854863Z [INF] BacktestEngine: Min Winner: 1,638.73 USDT
2021-03-18T14:10:53.854870Z [INF] BacktestEngine: Min Loser: -402.07 USDT
2021-03-18T14:10:53.854878Z [INF] BacktestEngine: Avg Loser: -1,753.25857143 USDT
2021-03-18T14:10:53.854885Z [INF] BacktestEngine: Max Loser: -3,099.37 USDT
2021-03-18T14:10:53.854892Z [INF] BacktestEngine: Win Rate: 0.07
2021-03-18T14:10:53.854899Z [INF] BacktestEngine: Expectancy: -1,527.126 USDT
2021-03-18T14:10:53.854906Z [INF] BacktestEngine: -----------------------------------------------------------------
2021-03-18T14:10:53.854914Z [INF] BacktestEngine: BTC
2021-03-18T14:10:53.854922Z [INF] BacktestEngine: -----------------------------------------------------------------
2021-03-18T14:10:53.854944Z [INF] BacktestEngine: PnL: 0.0 BTC
2021-03-18T14:10:53.854954Z [INF] BacktestEngine: PnL%: 0.0%
2021-03-18T14:10:53.854961Z [INF] BacktestEngine: Max Winner: 0.0 BTC
2021-03-18T14:10:53.854968Z [INF] BacktestEngine: Avg Winner: 0.0 BTC
2021-03-18T14:10:53.854975Z [INF] BacktestEngine: Min Winner: 0.0 BTC
2021-03-18T14:10:53.854982Z [INF] BacktestEngine: Min Loser: 0.0 BTC
2021-03-18T14:10:53.854989Z [INF] BacktestEngine: Avg Loser: 0.0 BTC
2021-03-18T14:10:53.854996Z [INF] BacktestEngine: Max Loser: 0.0 BTC
2021-03-18T14:10:53.855002Z [INF] BacktestEngine: Win Rate: 0.0
2021-03-18T14:10:53.855009Z [INF] BacktestEngine: Expectancy: 0.0 BTC
2021-03-18T14:10:53.855016Z [INF] BacktestEngine: -----------------------------------------------------------------
2021-03-18T14:10:53.855024Z [INF] BacktestEngine: Returns
2021-03-18T14:10:53.855031Z [INF] BacktestEngine: -----------------------------------------------------------------
2021-03-18T14:10:53.859395Z [INF] BacktestEngine: Annual return: -99.77%
2021-03-18T14:10:53.859419Z [INF] BacktestEngine: Cum returns: -2.39%
2021-03-18T14:10:53.859429Z [INF] BacktestEngine: Max drawdown: -2.39%
2021-03-18T14:10:53.859437Z [INF] BacktestEngine: Annual vol: nan%
2021-03-18T14:10:53.859444Z [INF] BacktestEngine: Sharpe ratio: nan
2021-03-18T14:10:53.859451Z [INF] BacktestEngine: Calmar ratio: -41.78
2021-03-18T14:10:53.859458Z [INF] BacktestEngine: Sortino ratio: nan
2021-03-18T14:10:53.859465Z [INF] BacktestEngine: Omega ratio: nan
2021-03-18T14:10:53.859472Z [INF] BacktestEngine: Stability: nan
2021-03-18T14:10:53.859478Z [INF] BacktestEngine: Returns Mean: -0.02388
2021-03-18T14:10:53.859485Z [INF] BacktestEngine: Returns Variance: 0.0
2021-03-18T14:10:53.859492Z [INF] BacktestEngine: Returns Skew: 0.0
2021-03-18T14:10:53.859499Z [INF] BacktestEngine: Returns Kurtosis: -3.0
2021-03-18T14:10:53.859505Z [INF] BacktestEngine: Tail ratio: 1.0
2021-03-18T14:10:53.859512Z [INF] BacktestEngine: Alpha: nan
2021-03-18T14:10:53.859519Z [INF] BacktestEngine: Beta: nan
2021-03-18T14:10:53.859527Z [INF] BacktestEngine:
2021-03-18T14:10:53.859535Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.859543Z [INF] BacktestEngine: LOG STORE
2021-03-18T14:10:53.859550Z [INF] BacktestEngine: =================================================================
2021-03-18T14:10:53.859558Z [INF] BacktestEngine: No log messages were stored.
balance_USDT balance_BTC
timestamp
2020-08-14 10:00:00.223000+00:00 1000.00000000 1.00000000
2020-08-14 10:22:34.574000+00:00 1000.00000000 1.00000000
2020-08-14 10:32:37.428000+00:00 -120.00000000 1.00000000
2020-08-14 10:32:37.428000+00:00 -120.00000000 1.00000000
2020-08-14 10:46:11.428000+00:00 -1370.00000000 1.00000000
2020-08-14 10:46:11.428000+00:00 -1370.00000000 1.00000000
2020-08-14 11:00:02.097000+00:00 -1930.00000000 1.00000000
2020-08-14 11:00:02.097000+00:00 -1930.00000000 1.00000000
2020-08-14 11:14:02.429000+00:00 -3370.00000000 1.00000000
2020-08-14 11:14:02.429000+00:00 -3370.00000000 1.00000000
2020-08-14 11:33:00.084000+00:00 -3580.00000000 1.00000000
2020-08-14 11:33:00.084000+00:00 -3580.00000000 1.00000000
2020-08-14 11:37:05.204000+00:00 -4480.00000000 1.00000000
2020-08-14 11:37:05.204000+00:00 -4480.00000000 1.00000000
2020-08-14 12:02:15.235000+00:00 -4310.00000000 1.00000000
2020-08-14 12:02:15.235000+00:00 -4310.00000000 1.00000000
2020-08-14 12:39:31.933000+00:00 -5230.00000000 1.00000000
2020-08-14 12:39:31.933000+00:00 -5230.00000000 1.00000000
2020-08-14 12:53:46.057000+00:00 -6330.00000000 1.00000000
2020-08-14 12:53:46.057000+00:00 -6330.00000000 1.00000000
2020-08-14 13:18:33.274000+00:00 -5880.00000000 1.00000000
2020-08-14 13:18:33.274000+00:00 -5880.00000000 1.00000000
2020-08-14 13:30:51.460000+00:00 -8130.00000000 1.00000000
2020-08-14 13:30:51.460000+00:00 -8130.00000000 1.00000000
2020-08-14 13:49:36.198000+00:00 -9950.00000000 1.00000000
2020-08-14 13:49:36.198000+00:00 -9950.00000000 1.00000000
2020-08-14 14:04:36.874000+00:00 -11290.00000000 1.00000000
2020-08-14 14:04:36.874000+00:00 -11290.00000000 1.00000000
2020-08-14 14:24:35.817000+00:00 -11640.00000000 1.00000000
2020-08-14 14:24:35.817000+00:00 -11640.00000000 1.00000000
2020-08-14 14:59:58.693000+00:00 -9150.00000000 1.00000000
order_id instrument_id side type quantity avg_price slippage timestamp
cl_ord_id
O-20200814-102234-000-001-1 1-001 ETH/USDT.BINANCE BUY MARKET 1000 424.58 0.0 2020-08-14 10:22:34.574000+00:00
O-20200814-103237-000-001-2 1-002 ETH/USDT.BINANCE SELL MARKET 1000 423.46 0.0 2020-08-14 10:32:37.428000+00:00
O-20200814-103237-000-001-3 1-003 ETH/USDT.BINANCE SELL MARKET 1000 423.46 0.0 2020-08-14 10:32:37.428000+00:00
O-20200814-104611-000-001-4 1-004 ETH/USDT.BINANCE BUY MARKET 1000 424.71 0.0 2020-08-14 10:46:11.428000+00:00
O-20200814-104611-000-001-5 1-005 ETH/USDT.BINANCE BUY MARKET 1000 424.71 0.0 2020-08-14 10:46:11.428000+00:00
O-20200814-110002-000-001-6 1-006 ETH/USDT.BINANCE SELL MARKET 1000 424.15 0.0 2020-08-14 11:00:02.097000+00:00
O-20200814-110002-000-001-7 1-007 ETH/USDT.BINANCE SELL MARKET 1000 424.14 0.0 2020-08-14 11:00:02.097000+00:00
O-20200814-111402-000-001-8 1-008 ETH/USDT.BINANCE BUY MARKET 1000 425.58 0.0 2020-08-14 11:14:02.429000+00:00
O-20200814-111402-000-001-9 1-009 ETH/USDT.BINANCE BUY MARKET 1000 425.57 0.0 2020-08-14 11:14:02.429000+00:00
O-20200814-113300-000-001-10 1-010 ETH/USDT.BINANCE SELL MARKET 1000 425.36 0.0 2020-08-14 11:33:00.084000+00:00
O-20200814-113300-000-001-11 1-011 ETH/USDT.BINANCE SELL MARKET 1000 425.35 0.0 2020-08-14 11:33:00.084000+00:00
O-20200814-113705-000-001-12 1-012 ETH/USDT.BINANCE BUY MARKET 1000 426.25 0.0 2020-08-14 11:37:05.204000+00:00
O-20200814-113705-000-001-13 1-013 ETH/USDT.BINANCE BUY MARKET 1000 426.25 0.0 2020-08-14 11:37:05.204000+00:00
O-20200814-120215-000-001-14 1-014 ETH/USDT.BINANCE SELL MARKET 1000 426.42 0.0 2020-08-14 12:02:15.235000+00:00
O-20200814-120215-000-001-15 1-015 ETH/USDT.BINANCE SELL MARKET 1000 426.43 0.0 2020-08-14 12:02:15.235000+00:00
O-20200814-123931-000-001-16 1-016 ETH/USDT.BINANCE BUY MARKET 1000 427.35 0.0 2020-08-14 12:39:31.933000+00:00
O-20200814-123931-000-001-17 1-017 ETH/USDT.BINANCE BUY MARKET 1000 427.35 0.0 2020-08-14 12:39:31.933000+00:00
O-20200814-125346-000-001-18 1-018 ETH/USDT.BINANCE SELL MARKET 1000 426.25 0.0 2020-08-14 12:53:46.057000+00:00
O-20200814-125346-000-001-19 1-019 ETH/USDT.BINANCE SELL MARKET 1000 426.26 0.0 2020-08-14 12:53:46.057000+00:00
O-20200814-131833-000-001-20 1-020 ETH/USDT.BINANCE BUY MARKET 1000 425.81 0.0 2020-08-14 13:18:33.274000+00:00
O-20200814-131833-000-001-21 1-021 ETH/USDT.BINANCE BUY MARKET 1000 425.81 0.0 2020-08-14 13:18:33.274000+00:00
O-20200814-133051-000-001-22 1-022 ETH/USDT.BINANCE SELL MARKET 1000 423.56 0.0 2020-08-14 13:30:51.460000+00:00
O-20200814-133051-000-001-23 1-023 ETH/USDT.BINANCE SELL MARKET 1000 423.57 0.0 2020-08-14 13:30:51.460000+00:00
O-20200814-134936-000-001-24 1-024 ETH/USDT.BINANCE BUY MARKET 1000 425.39 0.0 2020-08-14 13:49:36.198000+00:00
O-20200814-134936-000-001-25 1-025 ETH/USDT.BINANCE BUY MARKET 1000 425.38 0.0 2020-08-14 13:49:36.198000+00:00
O-20200814-140436-000-001-26 1-026 ETH/USDT.BINANCE SELL MARKET 1000 424.04 0.0 2020-08-14 14:04:36.874000+00:00
O-20200814-140436-000-001-27 1-027 ETH/USDT.BINANCE SELL MARKET 1000 424.04 0.0 2020-08-14 14:04:36.874000+00:00
O-20200814-142435-000-001-28 1-028 ETH/USDT.BINANCE BUY MARKET 1000 424.39 0.0 2020-08-14 14:24:35.817000+00:00
O-20200814-142435-000-001-29 1-029 ETH/USDT.BINANCE BUY MARKET 1000 424.39 0.0 2020-08-14 14:24:35.817000+00:00
O-20200814-145958-000-001-30 1-030 ETH/USDT.BINANCE SELL MARKET 1000 426.88 0.0 2020-08-14 14:59:58.693000+00:00
instrument_id strategy_id entry peak_quantity opened_time closed_time duration avg_open avg_close realized_points realized_return realized_pnl currency
position_id
P-20200814-102234-000-001-1 ETH/USDT.BINANCE 001 BUY 1000 2020-08-14 10:22:34.574000+00:00 2020-08-14 10:32:37.428000+00:00 0 days 00:10:02.854000 424.58 423.46 -1.12 -0.002638 -1968.04 USDT
P-20200814-103237-000-001-2 ETH/USDT.BINANCE 001 SELL 1000 2020-08-14 10:32:37.428000+00:00 2020-08-14 10:46:11.428000+00:00 0 days 00:13:34 423.46 424.71 -1.25 -0.002952 -2098.17 USDT
P-20200814-104611-000-001-3 ETH/USDT.BINANCE 001 BUY 1000 2020-08-14 10:46:11.428000+00:00 2020-08-14 11:00:02.097000+00:00 0 days 00:13:50.669000 424.71 424.15 -0.56 -0.001319 -1408.86 USDT
P-20200814-110002-000-001-4 ETH/USDT.BINANCE 001 SELL 1000 2020-08-14 11:00:02.097000+00:00 2020-08-14 11:14:02.429000+00:00 0 days 00:14:00.332000 424.14 425.58 -1.44 -0.003395 -2289.72 USDT
P-20200814-111402-000-001-5 ETH/USDT.BINANCE 001 BUY 1000 2020-08-14 11:14:02.429000+00:00 2020-08-14 11:33:00.084000+00:00 0 days 00:18:57.655000 425.57 425.36 -0.21 -0.000493 -1060.93 USDT
P-20200814-113300-000-001-6 ETH/USDT.BINANCE 001 SELL 1000 2020-08-14 11:33:00.084000+00:00 2020-08-14 11:37:05.204000+00:00 0 days 00:04:05.120000 425.35 426.25 -0.90 -0.002116 -1751.60 USDT
P-20200814-113705-000-001-7 ETH/USDT.BINANCE 001 BUY 1000 2020-08-14 11:37:05.204000+00:00 2020-08-14 12:02:15.235000+00:00 0 days 00:25:10.031000 426.25 426.42 0.17 0.000399 -682.67 USDT
P-20200814-120215-000-001-8 ETH/USDT.BINANCE 001 SELL 1000 2020-08-14 12:02:15.235000+00:00 2020-08-14 12:39:31.933000+00:00 0 days 00:37:16.698000 426.43 427.35 -0.92 -0.002157 -1773.78 USDT
P-20200814-123931-000-001-9 ETH/USDT.BINANCE 001 BUY 1000 2020-08-14 12:39:31.933000+00:00 2020-08-14 12:53:46.057000+00:00 0 days 00:14:14.124000 427.35 426.25 -1.10 -0.002574 -1953.60 USDT
P-20200814-125346-000-001-10 ETH/USDT.BINANCE 001 SELL 1000 2020-08-14 12:53:46.057000+00:00 2020-08-14 13:18:33.274000+00:00 0 days 00:24:47.217000 426.26 425.81 0.45 0.001056 -402.07 USDT
P-20200814-131833-000-001-11 ETH/USDT.BINANCE 001 BUY 1000 2020-08-14 13:18:33.274000+00:00 2020-08-14 13:30:51.460000+00:00 0 days 00:12:18.186000 425.81 423.56 -2.25 -0.005284 -3099.37 USDT
P-20200814-133051-000-001-12 ETH/USDT.BINANCE 001 SELL 1000 2020-08-14 13:30:51.460000+00:00 2020-08-14 13:49:36.198000+00:00 0 days 00:18:44.738000 423.57 425.39 -1.82 -0.004297 -2668.96 USDT
P-20200814-134936-000-001-13 ETH/USDT.BINANCE 001 BUY 1000 2020-08-14 13:49:36.198000+00:00 2020-08-14 14:04:36.874000+00:00 0 days 00:15:00.676000 425.38 424.04 -1.34 -0.003150 -2189.42 USDT
P-20200814-140436-000-001-14 ETH/USDT.BINANCE 001 SELL 1000 2020-08-14 14:04:36.874000+00:00 2020-08-14 14:24:35.817000+00:00 0 days 00:19:58.943000 424.04 424.39 -0.35 -0.000825 -1198.43 USDT
P-20200814-142435-000-001-15 ETH/USDT.BINANCE 001 BUY 1000 2020-08-14 14:24:35.817000+00:00 2020-08-14 14:59:58.693000+00:00 0 days 00:35:22.876000 424.39 426.88 2.49 0.005867 1638.73 USDT
For example, I would like to consider some simple functions (core\functions.pyx):
You are right that:
"> 10x faster than np.mean
"
"> 10x faster than np.std
"
(see performance tests: tests\performance_tests\test_perf_functions.py)
BUT ...
This is true when a list length < ~100
Using lists to pass values for calculation,e.g., mean and std, has significant drawback:
significant loss of speed calculation with increasing length (speed degradation).
(see tables below).
So my proposal - use np.ndarray
Annotation:
1. np_mean - 'mean' сomputation with numpy.mean
2. fast_mean - 'mean' сomputation with fast_mean (current nautilus_trader version - 1.113)
3. fast_mean_new - 'mean' сomputation with fast_mean_new (new version of fast_mean)
4. N samples - numbers of samples in the list/array
5. us = microsec
Numbers tests (TESTS) for each timing = 10000 - 1000 (dependent from N samples)
7. Test PC - CPU Type AMD Ryzen 7 1700, 3685 MHz
8. row "20; 11.8us (12.9); 2.5us (2.8); 0.9us (1.0)":
- 20 = N samples
- 11.8us (12.9) = average computation time with 'np_mean' (12.9 ~= 11.8/0.9)
- 2.5us ( 2.8) = average computation time with 'fast_mean' ( 2.8 ~= 2.5/0.9)
- 0.9us ( 1.0) = average computation time with 'fast_mean_new' ( 1.0 = 0.9/0.9) (standard=basis)
N samples | np_mean | fast_mean | fast_mean_new |
---|---|---|---|
5 | 10.6us (11.9) | 0.7us (0.8) | 0.9us (1.0) |
10 | 12.0us (13.3) | 1.4us (1.5) | 0.9us (1.0) |
20 | 11.8us (12.9) | 2.5us (2.8) | 0.9us (1.0) |
50 | 11.3us (14.4) | 4.7us (5.9) | 0.8us (1.0) |
100 | 11.0us (11.0) | 8.7us (8.7) | 1.0us (1.0) |
200 | 11.9us (11.7) | 17.7us (17.4) | 1.0us (1.0) |
500 | 10.5us (7.6) | 42.0us (30.4) | 1.4us (1.0) |
1000 | 11.4us (6.3) | 82.7us (45.6) | 1.8us (1.0) |
2000 | 12.4us (4.4) | 162.5us (57.4) | 2.8us (1.0) |
5000 | 13.1us (2.3) | 449.3us (79.3) | 5.7us (1.0) |
10000 | 14.2us (1.4) | 837.3us (83.6) | 10.0us (1.0) |
20000 | 17.2us (0.9) | 1675.0us (84.4) | 19.8us (1.0) |
50000 | 28.2us (0.6) | 4182.1us (92.8) | 45.1us (1.0) |
100000 | 42.7us (0.5) | 8287.9us (94.3) | 87.9us (1.0) |
1000000 | 816.3us (0.9) | 84214.4us (93.2) | 903.9us (1.0) |
10000000 | 10978.4us (1.2) | 882933.4us (96.8) | 9124.7us (1.0) |
Annotation:
1. np_std - 'std' сomputation with numpy.std
2. fast_std - 'std' сomputation with fast_std (current nautilus_trader version - 1.113)
3. fast_std_new - 'std' сomputation with fast_std_new (new version of fast_std)
4. N samples - ...
5. us = ...
7. Test PC = ...
8. ...
N samples | np_std | fast_std | fast_std_new |
---|---|---|---|
5 | 33.6us (25.4) | 1.8us (1.3) | 1.3us (1.0) |
10 | 37.2us (27.9) | 3.4us (2.5) | 1.3us (1.0) |
20 | 34.1us (30.1) | 6.8us (6.0) | 1.1us (1.0) |
50 | 34.2us (29.4) | 13.4us (11.5) | 1.2us (1.0) |
100 | 33.9us (24.7) | 26.3us (19.1) | 1.4us (1.0) |
200 | 37.1us (21.8) | 51.9us (30.6) | 1.7us (1.0) |
500 | 34.9us (15.7) | 132.8us (59.9) | 2.2us (1.0) |
1000 | 37.6us (12.7) | 254.9us (86.2) | 3.0us (1.0) |
2000 | 42.5us (8.1) | 531.7us (102.0) | 5.2us (1.0) |
5000 | 48.5us (4.7) | 1336.0us (129.5) | 10.3us (1.0) |
10000 | 57.6us (2.8) | 2630.5us (126.6) | 20.8us (1.0) |
20000 | 67.6us (1.8) | 5192.6us (141.0) | 36.8us (1.0) |
50000 | 100.4us (1.1) | 13126.2us (147.7) | 88.9us (1.0) |
100000 | 149.8us (0.8) | 27235.6us (151.4) | 179.8us (1.0) |
1000000 | 4524.3us (2.5) | 267034.9us (147.7) | 1807.9us (1.0) |
10000000 | 50570.1us (2.8) | 2748350.5us (151.2) | 18173.4us (1.0) |
If we don't consider the special cases (using increased computation accuracy, alg Kahan and ... ,
and parallel computations) I propose new fast versions for 'fast_mean', 'fast_std' and other functions
(Note: no problem to convert, for example, deque to nd.array ...)
Send you all code today later.
(with new version for tests\performance_tests\test_perf_functions.py)
Account should reflect the equity balance at all times. In case there is a position open, account.equity should be something like cash + position_value_unleveraged + unrealized_pnl
DISCALIMER: I'm filing these issues as I find them. Have a few more but trying to file one at a time. This was noticed in the master branch about 2 weeks (10/24/2020) ago so the code below is not completely up to date. Please disregard if this issue has already been resolved.
I haven't been able to fully identify the issue here but wanted to bring it to your attention. Some strange things are occurring in TickDataWrangler.pre_process() due to the way that zero-volume ticks are removed.
As an example, I added some extra debug prints using the included GBPUSD 1-minute data which is aggregated to create 3-minute bars. In some instances a tick may have zero-volume and gets removed. The bars created from these ticks do not seem to handle the removed ticks properly.
2020-10-24T08:21:59.562Z [WRN] BacktestEngine: 2008-01-01 04:49:00+00:00 Generating tick...
2020-10-24T08:21:59.562Z [WRN] BacktestEngine: 2008-01-01 04:49:00+00:00 Received tick ---> "GBP/USD.SIM,1.98500,1.98505,0,0,2008-01-01T04:49:59.900Z"
2020-10-24T08:21:59.562Z [WRN] BacktestEngine: 2008-01-01 04:49:59.900000+00:00 Advanced clock to tick.timestamp "2008-01-01 04:49:59.900000+00:00"
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:49:59.900000+00:00 Generating tick...
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:49:59.900000+00:00 Received tick ---> "GBP/USD.SIM,1.98500,1.98505,0,0,2008-01-01T04:49:59.900Z"
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:49:59.900000+00:00 Advanced clock to tick.timestamp "2008-01-01 04:49:59.900000+00:00"
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:49:59.900000+00:00 Generating tick...
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:49:59.900000+00:00 Received tick ---> "GBP/USD.SIM,1.98488,1.98493,0,0,2008-01-01T04:49:59.900Z"
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:49:59.900000+00:00 Advanced clock to tick.timestamp "2008-01-01 04:49:59.900000+00:00"
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:49:59.900000+00:00 Generating tick...
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:49:59.900000+00:00 Received tick ---> "GBP/USD.SIM,1.98488,1.98493,24,24,2008-01-01T04:50:00.000Z"
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:50:00+00:00 Advanced clock to tick.timestamp "2008-01-01 04:50:00+00:00"
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:50:00+00:00 Generating tick...
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:50:00+00:00 Received tick ---> "GBP/USD.SIM,1.98495,1.98500,0,0,2008-01-01T04:51:59.900Z"
2020-10-24T08:21:59.563Z [WRN] BacktestEngine: 2008-01-01 04:51:59.900000+00:00 Advanced clock to tick.timestamp "2008-01-01 04:51:59.900000+00:00"
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: Received GBP/USD.SIM-3-MINUTE-BID Bar(1.98505,1.98520,1.98488,1.98488,180,2008-01-01T04:51:00.000Z)
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: [CMD]--> SubmitOrder(venue=SIM, trader_id=BACKTESTER-000, account_id=SIM-000-SIMULATED, cl_ord_id=O-20080101-045159-000-GBPUSD-20, position_id=B-GBP/USD-10, strategy_id=EMACross-GBPUSD.
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: <--[EVT] OrderSubmitted(account_id=SIM-000-SIMULATED, cl_ord_id=O-20080101-045159-000-GBPUSD-20, id=bf8f8b2a-157a-0aff-8476-98e84a95cb28).
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: <--[EVT] OrderAccepted(account_id=SIM-000-SIMULATED, cl_ord_id=O-20080101-045159-000-GBPUSD-20, order_id=B-GBP/USD-20, id=bf8f8b2a-157a-0aff-8476-98e84a95cb28).
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: <--[EVT] OrderFilled(account_id=SIM-000-SIMULATED, cl_ord_id=O-20080101-045159-000-GBPUSD-20, order_id=B-GBP/USD-20, position_id=B-GBP/USD-10, strategy_id=EMACross-GBPUSD, symbol=GBP/USD.SIM, side=SELL-TAKER, filled_qty=100,000, leaves_qty=0, avg_price=1.98495, commission=10.00 GBP, id=bf8f8b2a-157a-0aff-8476-98e84a95cb28).
2008-01-01T04:51:59.900Z [INF] Portfolio: GBP/USD.SIM net position = 0
2008-01-01T04:51:59.900Z [INF] Portfolio: Updated SIM position maintenance margin to 0.00 USD
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: <--[EVT] PositionClosed(account_id=SIM-000-SIMULATED, position_id=B-GBP/USD-10, strategy_id=EMACross-GBPUSD, entry=BUY, duration=00:03:59.900000, avg_open=1.98509, avg_close=1.98495, realized_points=-0.00014, realized_return=-0.007%, realized_pnl=2.95 GBP, id=bf8f8b2a-157a-0aff-8476-98e84a95cb28).
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: [CMD]--> SubmitOrder(venue=SIM, trader_id=BACKTESTER-000, account_id=SIM-000-SIMULATED, cl_ord_id=O-20080101-045159-000-GBPUSD-21, position_id=NULL, strategy_id=EMACross-GBPUSD.
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: <--[EVT] OrderSubmitted(account_id=SIM-000-SIMULATED, cl_ord_id=O-20080101-045159-000-GBPUSD-21, id=bf8f8b2a-157a-0aff-8476-98e84a95cb28).
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: <--[EVT] OrderAccepted(account_id=SIM-000-SIMULATED, cl_ord_id=O-20080101-045159-000-GBPUSD-21, order_id=B-GBP/USD-21, id=bf8f8b2a-157a-0aff-8476-98e84a95cb28).
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: <--[EVT] OrderFilled(account_id=SIM-000-SIMULATED, cl_ord_id=O-20080101-045159-000-GBPUSD-21, order_id=B-GBP/USD-21, position_id=B-GBP/USD-11, strategy_id=EMACross-GBPUSD, symbol=GBP/USD.SIM, side=SELL-TAKER, filled_qty=100,000, leaves_qty=0, avg_price=1.98495, commission=10.00 GBP, id=bf8f8b2a-157a-0aff-8476-98e84a95cb28).
2008-01-01T04:51:59.900Z [INF] Portfolio: GBP/USD.SIM net position = -100000
2008-01-01T04:51:59.900Z [INF] Portfolio: Updated SIM position maintenance margin to 0.00 USD
2008-01-01T04:51:59.900Z [INF] EMACross-GBPUSD: <--[EVT] PositionOpened(account_id=SIM-000-SIMULATED, position_id=B-GBP/USD-11, strategy_id=EMACross-GBPUSD, entry=SELL, avg_open=1.98495, SHORT 100,000 GBP/USD.SIM, id=bf8f8b2a-157a-0aff-8476-98e84a95cb28).
2020-10-24T08:21:59.564Z [WRN] BacktestEngine: 2008-01-01 04:51:59.900000+00:00 Generating tick...
2020-10-24T08:21:59.564Z [WRN] BacktestEngine: 2008-01-01 04:51:59.900000+00:00 Received tick ---> "GBP/USD.SIM,1.98505,1.98510,0,0,2008-01-01T04:51:59.900Z"
2020-10-24T08:21:59.564Z [WRN] BacktestEngine: 2008-01-01 04:51:59.900000+00:00 Advanced clock to tick.timestamp "2008-01-01 04:51:59.900000+00:00"
2020-10-24T08:21:59.564Z [WRN] BacktestEngine: 2008-01-01 04:51:59.900000+00:00 Generating tick...
2020-10-24T08:21:59.564Z [WRN] BacktestEngine: 2008-01-01 04:51:59.900000+00:00 Received tick ---> "GBP/USD.SIM,1.98480,1.98485,0,0,2008-01-01T04:51:59.900Z"
2020-10-24T08:21:59.564Z [WRN] BacktestEngine: 2008-01-01 04:51:59.900000+00:00 Advanced clock to tick.timestamp "2008-01-01 04:51:59.900000+00:00"
2020-10-24T08:21:59.564Z [WRN] BacktestEngine: 2008-01-01 04:51:59.900000+00:00 Generating tick...
2020-10-24T08:21:59.564Z [WRN] BacktestEngine: 2008-01-01 04:51:59.900000+00:00 Received tick ---> "GBP/USD.SIM,1.98490,1.98495,253,253,2008-01-01T04:52:00.000Z"
2020-10-24T08:21:59.564Z [WRN] BacktestEngine: 2008-01-01 04:52:00+00:00 Advanced clock to tick.timestamp "2008-01-01 04:52:00+00:00"
Note how a tick with timestamp 04:51.59.900 prints which then triggers a bar with timestamp 04:51:00.000 (i.e. clock is moved backwards, which probably should not be allowed).
These ticks were removed by TickDataWrangler because bid_volume+ask_volume==0 but they were not
removed from bar data in data_client since volume is sum of 3 bars and != 0. It seems that adding a check to make sure all events occur in sequential order (i.e. the timestamp of a new event must be >= the current timestamp) would at least catch this issue and throw an error.
Apologize for not providing code to reproduce but I can put something together if this doesn't immediately explain the issue.
Position unrealized PNL with wrong value after executing an order of size grater than position size in the opposite direction.
I opened a PR with a test illustrating this:
#59
I'm not sure what the expected behaviour should be in this case, I'm thinking that flattening the current position and opening a new one in the opposite direction should be the right approach. Another option is to keep the same position and reset the impacted fields, but then the realized_pnl will not contain the PNL prior to changing the direction.
Currently there are some properties for margin reported in the AccountState
event. The Account
also tracks some margin information, however margin and leverage is not being entirely simulated.
Proposing we introduce some leverage configuration options and accurately simulate margin inside SimulatedMarket
. Orders should be rejected, and positions liquidated if margin requirements are not met - as appropriate for a generic exchange at this stage.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.