`backtest()`: Retrain every n steps about darts HOT 11 CLOSED

Thanks for pointing this out @aschl. We are currently re-working quite substantially the backtesting functionality, and indeed our plan is to introduce a stride parameter like you suggest. We will also introduce moving windows for the training set (instead of expending only, as now). These should come in a release sometime soon.

from darts.

Hello @aschl, while we don't have a date yet for the release of this feature. If this something you would like to try as soon as possible the stride parameter has already been added to the backtest_forecasting method on the develop branch. Keep in mind that we are currently working on a quite heavy refactor of the whole backtesting.py file and the API for backtesting might change in the upcoming release.

from darts.

I checked backtesting.py in the develop branch. If I understand it correctly, the stride parameters determines the distance between two consecutive predictions. This might be helpful if e.g. you only want to predict January values (and skip Feb-Dec forecast). I think there are quite a few use cases when you need all prediction points but would like to train the model only on every x time steps. This would increase efficiency quite a lot without necessarily loosing predictive power (adding a few more samples to a series of few hundert observations doesn't substantially improve the model if the distribution remains similar).

The idea behind scikitlearn's TimeSeriesSplit() is exactly this. You split the time series in k cross-validation pieces and retrain the model only k times. This way you obtain a relatively unbiased estimate of the forecasting error with reasonable computational efforts.

(I see that you implemented a retrain argument, which goes into this direction. This would however only allow to either retrain for each prediction or only once with the first sample.)

from darts.

Yes. Exactly.
I am currently busy with some other projects. Not sure when I will find time to come back to this. Please keep me updated.

from darts.

Can I take at this one up if this is still open?

from darts.

You're right @deltamacht. Gridsearch is still pretty basic... There's however a stride parameter in gridsearch() which maybe goes some way into doing what you're looking for.

from darts.

Awesome! Looking forward to seeing this feature. The idea of incorporating the moving window is also great and very useful.
When do you approximately plan to release this?

from darts.

Hi @aschl and sorry for the delay answering your last comment. If I understand correctly, your suggestion would be to replace retrain:bool=True with something like retrain_every_n:int = 1, am I correct ?

If that's the case, I think you make a good point. Feel free to contribute if you can! Otherwise I'll put it on our radar.

from darts.

@pravarmahajan sorry for the late reply. We would be very happy to receive a PR to introduce a "retrain every N steps" feature to historical forecasting / backtesting!

from darts.

Just looping back to this. As I understand, #1139 addressed the concern on retraining every n steps in the retrain behavior in backtest(), but this parameter isn't exposed in the gridsearch method. With regards to the discussion above about having some behavior that would be similar to Sklearn's TimeSeriesSplit, am I correct in thinking that this type of cross validation isn't easily specified in the gridsearch method at present?

from darts.

Implemented in v0.22.0 🚀

from darts.