MarketData <> prepare_data about cvxportfolio HOT 7 CLOSED

BTW, the default UX will be like what is in the examples now, i.e.

results = MarketSimulator(['AAPL', 'GOOG', 'AMZN']).backtest([strategy1, strategy2], '2000-01-01')
print(results)
results.plot()

(This is not the syntax now.)

from cvxportfolio.

No, is the other way round, right now only the methods inside marketsimulator are being used and I'm building outside of it their replacements with some improvements. Everything is tested (both old and new). I didn't touch the old pieces as I am building the new so no disruption there. The new pieces (marketdata etc.) are 100% tested but not currently plugged in the main api. Git branches are an overkill for this project at the moment.

from cvxportfolio.

Also, sorry I didn't mean to sound dismissive of your comment, you're right that it's not great to have code duplication in the master branch (I wrote the new stuff on Monday and I thought to merge earlier but will probably have to wait for the weekend). I can share what I have in mind. MarketSimulator.__init__ takes a list of cost functions (currently I wrote classes that implement a single method but will be simple callbacks instead) and **kwargs that are passed to them (e.g., linear_transaction_cost, cash_borrowing_spread, ...). MarketData holds and serve data to both the simulator and and the policy and is also initialized by the simulator with the optional **kwargs. MarketSimulator becomes a very thin class almost stateless that implements the simulate and backtest methods, also BacktestResult will change and take on a few things that are currently done inside MarketSimulator. Main issue is about multiprocessing, it's very tricky to pass classes with lots of internal state through the interprocess communication system (they all get pickled and unpickled on the other end) so I need to make sure that classes have as little state as possible. In fact MarketSimulator.backtest will only implement logic to initialize a backtest (and break it up into smaller backtests as the universe changes with IPOs or bankruptcies, cfr. #85) and then call a separate _backtest function, which is the one that is sent to multiprocessing. Again, the goal is to have a final product that is both easy to use (expose to the user a clear object model, not a bunch of functions), easy to extend (user-defined extensions should be easy to write) and easy to read and debug. Also, since everything goes through multiprocessing, at the lower level it should ideally call functions or near-stateless classes' methods).

from cvxportfolio.

Ah, it's my bad then, I didn't realize that MarketData was new and should have checked the history not just the current code.
It's just that things like MarketData being unused confused me.
I think the proposed class structure (MarketData servicing both the policy and the simulator) makes a lot of sense. And can perhaps solve the problem with nan assets (#85) in the same go.
I will continue to get up to speed with the codebase.

from cvxportfolio.

Follow-up question on multiprocessing - can you help me understand how you envision a fully multiprocessing compatible backtest, as it was my understanding that backtesting on time t requires state knowledge of holdings for t-1?

from cvxportfolio.

Multiprocessing is used when doing multiple backtests (which will be largely automated once we make the hyperparameter optimization as part of the simulator logic). This was already implemented in old cvxportfolio (branch 0.0.X) and used in the example notebooks. It gets trickier with thicker classes, which is the reason why I'm refactoring the simulator logic now. Also we used to use an external multiprocess library, instead I moved to the standard library multiprocessing module which is less forgiving (but with better support). The problem with multiprocessing is that is hard to debug, hence code needs to be crystal clear. You can't really parallelize a single backtest because it is path-dependent (even without tcosts). All cvx* stack is mostly single-threaded (apart from a few BLAS level3 / Lapack calls which are very rare) so multiprocessing is a great fit. Also I'm not closing some old issues because they contain stuff that needs to be processed (e.g., in this one there is a lot of text that needs to go in the docs).

from cvxportfolio.

BTW, the above is true for optimization-based strategies, for toy strategies easier parallelization might work. Everything in cvxportfolio maps the content of the book, the dynamics is explained in Chapter 2. The dynamics and parallelization are the same as they have always been in cvxportfolio since the initial 2016 release.

from cvxportfolio.