Error ''Score difference between baseline and instance ... is too low < 0.1.'' about timeshap HOT 2 CLOSED

Hello @bryancoh ,

Regarding point 1, you are correct. This issue will be fixed in the next push to the repository. Thank you very much for letting us know.

Regarding point 2, the value mentioned was empirically found to work well with our financial dataset. Since we were dealing with a highly unbalanced dataset, the average score for our instances was very low. In our explanations, we focused only on the positive instances, ensuring a significant difference between the baseline and explained instances.

The issue you mentioned has been raised in several other cases, and we are actively working on a solution that will benefit everyone. The main objective of the warning is to prevent explanations that may be derived solely from noise. TimeSHAP (and KernelSHAP) divide the difference in score between the background instance and the explained instance among the features. If this score is too low, there are two possible scenarios: either some features have a substantial positive contribution while others have a substantial negative contribution that cancel each other out, or all explanations have very low values that can be easily affected by the final linear fitting of the SHAP method. The implementation of this check was intended to prevent errors in the second case.

Currently, there is no direct implemented way for users to change the threshold value or entirely skip the check. If you desire this behavior, you would need to modify the source code of TimeSHAP. However, based on the feedback we have received, we are considering implementing two features. We would appreciate your input on these:

1. Allowing users to define the threshold value instead of it being fixed at 0.1.
2. Providing an option to skip the check if the user desires.

I hope this answer is helpful. If you have any further questions feel free to ask.

from timeshap.

Thank you for your answer, I will modify the source code for now

from timeshap.