Comments (2)
Hi there, that's an interesting situation. When you try a different train-test split, do you train a new model? Or do you use a different train-test split (with the same model) just when estimating SAGE values? And also, is the estimator running to convergence so that you get pretty narrow confidence intervals?
Assuming that the SAGE values are known with high confidence (narrow confidence intervals), here's what I think you can do.
If it's the first situation, then it may mean that your model depends quite a bit on the train-test split. Ideally that wouldn't happen, especially if there's enough data, but averaging the SAGE values is a reasonable approach. (For the confidence intervals, I would calculate the standard deviations by taking the square root of the average variance.)
If it's the second situation, then I would put more trust in the SAGE values that are calculated using data that was not touched during training (the test data), because the loss values (and therefore the SAGE values) may be artificially changed by overfitting to the train set.
Let me know how that sounds.
from sage.
Hello, thank you for the answer! It is the first situation. Maybe there is not enough data. I guess I will average values and calculate confidence intervals as you suggest. Thanks!
from sage.
Related Issues (16)
- License HOT 1
- Parallelized computation HOT 15
- Possibility to use presegmented images HOT 3
- Exception encountered when calling layer "gru" (type GRU). HOT 1
- TreeSAGE ? HOT 6
- Zero-One Loss in Classification HOT 4
- Unstable SAGE values HOT 4
- All negative SAGE values in classification HOT 5
- Mismatch between feature importancies from `GroupedMarginalImputer` and `MarginalImputer` HOT 5
- pip install HOT 2
- Explanation about new changes in the SAGE package and addition of Model sensitivity module. HOT 6
- PermutationEstimator runs infinitely when gap = 0 HOT 3
- Shape mismatch on XGB.Classifier HOT 3
- adaptive estimator for online data HOT 2
- SAGE with NLP/Huggingface HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sage.