Comments (6)
Hi Achilleas,
thank you for your feedback and happy to hear that ppscore is helpful to you.
Currently, you cannot change the classification score from F1 to ROC but I agree that this would be a useful addition.
How would you like to change the score from F1 to ROC? How would you like the user experience/API to look like?
Regards,
Florian
from ppscore.
I am not sure, I'm a programming noob and still learning the ropes with ML. I'm a neuroscientist... Also, I've done some reading and different cross-validations depending on the problem and thus for my small dataset I've changed the stratified cv that PPS score uses to leave one out CV (LOOCV). That's all good and done and easy to change and apply on PPS.
But then I've read that different performance metrics are used depending on the morphology of the dataset. E.g., different metrics should be used according to whether you are trying to solve a multiclass problem or a binary classification problem. Then different metrics should be used depending on the class distribution. For example 'accuracy' is generally bad (as you are aware) but 'balanced accuracy' might be better than even an 'F1' score for a binary classification problem with imbalanced classes. In general, there are various types of accuracy metrics that have different strength and weaknesses depending on the dataset and what you are trying to achieve. It would be good to allow the user to easily choose which one they want e.g.,
pps_predictors_results= pps.predictors(dataset, y= "Group", random_seed = 16, cross_validation=LOOCV_method, scoring ='balanced_accuracy')
The scoring variable could have different accuracy metrics taken from sklearn e.g., 'accuracy', 'adjusted_rand_score', 'average_precision', 'balanced_accuracy', 'f1', 'f1_macro', 'f1_micro', 'f1_samples', 'f1_weighted', 'neg_log_loss', 'precision' etc. (suffixes apply as with ‘f1’), 'recall' etc. (suffixes apply as with ‘f1’), ‘jaccard’ etc. (suffixes apply as with ‘f1’), 'roc_auc', ‘roc_auc_ovr’, ‘roc_auc_ovo’, ‘roc_auc_ovr_weighted’, ‘roc_auc_ovo_weighted’
Am I making sense? Like I said I'm quite new to ML. I've only started learning ML a year ago.
from ppscore.
I like the suggestion and think that this can be worthwhile.
About the API proposal:
- we might need to find a way to differentiate between regression and classification metrics (in case of supporting this within predictors)
- regarding the
cross_validation=LOOCV_method
was it on purpose that you refer to the LOOCV_method object that might be defined somewhere or did you want to refer to it via a string? - About adding other evaluation metrics: for all of them we need to define the limits in terms of best case and bottom baseline. In some cases like the F1 this might not always be so trivial
Also, would you like to take on this task?
from ppscore.
- Cool.
- No I defined LOOCV_method before
- Yes, I agree. Choosing the right performance metric depends on the data.
I wish I could but I don't have the technical knowledge to do this I'm afraid!
from ppscore.
Alright, thank you for the answers.
And no worries if you cannot work on it yourself - the suggestion and rough desgn of the user experience is also very valuable.
Also, maybe you will rise to the occasion at some point - after all, we grow with our challenges :)
Take care and talk to you soon
from ppscore.
I would also like a better approach to handling class imbalance and am happy to have a go at adding this functionality. I was thinking of making the setting of the average
parameter in the f1_score as a user optional input instead of as a hardcoded value of "weighted". This would be a pretty lightweight approach and give you all the functionality and error handling of the underlying sklearn function. How does this sound @FlorianWetschoreck ?
from ppscore.
Related Issues (20)
- Data preprocessing and information leakage HOT 14
- [SUGGEST] Release a verson supported GPU HOT 2
- ppscore when model_score>baseline_score HOT 3
- There should be an option to override the attribute type like PyCaret HOT 4
- Scikit-learn dependency < 1.0.0 HOT 14
- [Suggestion]: Plot the Decision Tree for pps.score HOT 3
- Readme / docs unclear about using ppscore on time series data HOT 3
- pytests failing with pandas==1.4.0 HOT 1
- Thought on a possible enhancement of the PPS HOT 2
- What does PPS score? HOT 4
- Add support to release Linux aarch64 wheels HOT 4
- Cannot install ppscore HOT 1
- Your package isn't compatible with scikit-learn 1.0.1 HOT 2
- How to report PPS HOT 1
- Question About Data Order HOT 12
- y predicted values given x HOT 3
- Performance HOT 3
- differnt baseline scores for the same y HOT 1
- How to deal with heavy imbalanced data? For example, when the target is 99 "negative" to 1 "positive"
- pandas >2 support HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ppscore.