Understanding LOFO Importance about lofo-importance HOT 4 CLOSED

Aha ok. So I learned two things: Checking for correlated features before applying LOFO and not to ignore the standard deviation in the LOFO result dataframe :) Also, thank you for the additional links. I guess this is closed now. Thanks.

from lofo-importance.

Hi @r0f1, thanks for letting us know. As far as I see, the feature importances are very low and their standard deviations are relatively high. This means that positive ones are not significantly positive and negative ones are not significantly negative. Can you also make the same test by providing a model you know (instead of default model) which has high AUC on this dataset?

from lofo-importance.

Hi,
So you would recommend not to drop features if the standard deviation is too high?

I ran a couple of experiments on the BRCA dataset. I found that the default RandomForestClassifier has a good accuracy (0.912) and roc_auc (0.989). Running the LOFO importance with this model gave me unexpected results: Almost all features are in the red.

In order to do my feature selection procedure (=take only those features whose mean is above zero, irgnoring standard deviation), I switched to accuracy. Running the LOFO importance with accuracy, over halt the features were green, and others were red. I removed the red ones and re-ran the LOFO imporance on the green ones only, and then there were again some red ones.

Also, between the runs, the ordering of the features changed.

For fun, I calculated the permutation importance also, and the orderings a completely different from the LOFO importance orderings. Are these the results of random variability? Is there maybe another dataset that you would recommend to re-run those experiments?

Code to reproduce everything:
repo, binder

from lofo-importance.

Thanks for the extensive explanation. I have just checked the dataset. Almost all the features are over 0.9 correlated with another feature. Therefore removing one feature out doesn't have any significant effect. This is the reason why LOFO has small values as output with relatively high standard deviations. So while reading LOFO's output, please always consider the standard deviations.

Some dataset examples:
https://www.kaggle.com/divrikwicky/pf-f-lofo-importance-on-adversarial-validation
https://www.kaggle.com/divrikwicky/lofo-importance
https://www.kaggle.com/divrikwicky/santanderctp-lofo-feature-importance

from lofo-importance.