[ENH] Access the actual features (including lagged y) used as input for the model inside a reduction forecaster about sktime HOT 10 OPEN

Just out of my curiosity, can I ask how are you planning to use Shapley values here?

For example, strategy="direct" will use multiple models, and "actuals" are going to vary for different forecast horizons with strategy="recursive" if I understand correctly (in the sense fh=1 will only use last window_length training observations, fh=2 will use last window_length - 1 training observations and prediction for fh=1 and so on).

So, what shall you pass as "actual" values to shap?

(As said, it's not related to your issue, but out of my curiosity as I am also planning to have some explainability component for my office work.)

from sktime.

In my current application I care most about a few horizons only (1-3), and later horizons are increasingly less relevant. For now I'm simply planning to just look at the values for the first few horizons separately.

I'm aware of the issues with features varying by horizon in the recursive case, I'm not sure what the best solution is. Maybe offering a method to recreate the input for a given horizon is easier than storing it?

from sktime.

Related, this request by @yarnabrina to access the internal data in two-step exogenous forecast:
#5598

I wonder whether there should be a programmatic way to access internal preprocessed data.

from sktime.

From a design perspective, we could "dump" the formatted data in a _X-like argument, although we should be cautious as this can blow up the pickle size etc.

A "nicer" way would be to also allow the forecaster to act as a transformer, which is now possible with the object_type tag that can have multiple types. This was introduced to allow polymorphism for the graphical pipeline, FYI @benHeid.

There used to be a transform method, so I wonder whether this can simply be reactivated.

from sktime.

On a related note, there is a transformer (not much used afaik) which also addresses the issue, the ReducerTransform, which could be used for shapley values.

from sktime.

From a design perspective, we could "dump" the formatted data in a _X-like argument, although we should be cautious as this can blow up the pickle size etc.

A "nicer" way would be to also allow the forecaster to act as a transformer, which is now possible with the object_type tag that can have multiple types. This was introduced to allow polymorphism for the graphical pipeline, FYI @benHeid.

There used to be a transform method, so I wonder whether this can simply be reactivated.

I'd be fine with either, although I'm not sure how "dumping" would look like with the recursive reducer.

from sktime.

On a related note, there is a transformer (not much used afaik) which also addresses the issue, the ReducerTransform, which could be used for shapley values.

Thanks for pointing this out, I'll have a look. Does this mean the API reference on the homepage is incomplete? I've looked through the list of transformers, and this one isn't listed.

from sktime.

Does this mean the API reference on the homepage is incomplete

Yes, sorry.

The issue is, we are not sure how to test "estimator is not present on API reference page".
It should be picked up by the all_estimators utility though, since that is programmatic and crawls the package.

I've added it, and the new direct reducer prototype to the API reference:
#5690

from sktime.

although I'm not sure how "dumping" would look like with the recursive reducer.

I think that is precisely @yarnabrina's question, i.e., what should it even do.

from sktime.

although I'm not sure how "dumping" would look like with the recursive reducer.

I think that is precisely @yarnabrina's question, i.e., what should it even do.

Maybe just dump _X in a dictionary by horizon? Or provide a method that takes horizon as argument and returns X after fitting.

Thanks for updating the docs!

from sktime.