adefazio / classifierplots Goto Github PK

I was linked to this by @BrendanVR. Nice package! I was wondering if you would consider exporting the stuff in metrics.R? There is a real lack of support for scoring binary classifiers in modelr and these functions would be a good start! Unless you think they belong in a separate package for this purpose?

I've encountered an issue (using version 1.4.0) where the ROC curve is missing from the plot produced by roc_plot(). See the image below for an example. As far as I can tell no errors or warnings are produced in the process.

Classifier plots roc curve drawing appears to have visual artefacts when scores aren't spread out well (there are a large number of instances with the same score, leading to empty deciles).

The individual pieces that make up a classifierplot object render fine when you add them in an Rmd chunk, but when you include a the actual classifierplots function, it just prints out the message.

This works

density_plot(test_dat$Outcome, 
                        test_dat$PredEns)

This does not

grobEns <- classifierplots(test_dat$Outcome,
                        test_dat$PredEns) 
grobEns

And neither does this

gridExtra::grid.arrange(grobEns, 
                                top=textGrob("Ensemble model",
                      gp=gpar(fontsize=16,font=1)))

[1] "Calculating AUC ..."
[1] "(AUC) Sorting data ..."
[1] "(AUC) Calculating ranks ..."
[1] "AUC: 72.3611244722356"
[1] "Bootstrapping ROC curves"
[1] "Eval AUC"
[1] "Producing ROC plot"
[1] "Generating score density plot"
Error in alpha * 255 : non-numeric argument to binary operator

breaking_example_classifierplots.csv.gz

The manuals are poorly written, may I say, to the point of being non-existent...

In the calibration.R, qbeta function was used to calculate true probability in the calibration_plot, such as "qbeta(c(llb=0.025, lb=0.25, y=0.5, ub=0.75, uub=0.965), 0.5+positive, 0.5+bucket_size-positive)". Sorry I can't understand that. Could you please provide some expanations or some papers.
Thank you!

y_test_class = [0,0,0,0,0]
prob_class = [0.01,0.02,.45,0.36,0.001]
classifierplots(y_test_class,prob_class)
"""
Error in check_classifier_input_and_init(test.y, pred.prob) :
test.y had more than 2 unique values: 1
"""

An initial look into this shows a few issues. The tar command references a folder in a directory structure that's different to the R.template assumed layout. Also, library(devtools) (as per R.profile) doesn't exist in CI.

Does this repo require artefacts to be uploaded

Or

Should the scripts be corrected to work propery

@adefazio @HuwCampbell

Hi, I just installed the CRAN version.
I train a classifier using various packages and when I finally want to call classifierplots I get the error

alpha level NA, not in [0,1]

I can not share my whole code and data here. If I run the base example after doing my calculations I get the error too.
So there might be an incompatibility with the packages already loaded.

Do you have any idea if you look at the list of packages loaded.
Is there anything I can check?
Thank you!

library(classifierplots)
classifierplots(example_predictions$test.y, example_predictions$pred.prob)

and got the error:

[1] "Calculating AUC ..."
[1] "(AUC) Sorting data ..."
[1] "(AUC) Calculating ranks ..."
[1] "AUC: 90.5603213507625"
[1] "Bootstrapping ROC curves"
[1] "Eval AUC"
[1] "Producing ROC plot"
[1] "Generating score density plot"
Error in grDevices::rgb(col[1, ], col[2, ], col[3, ], alpha) :
alpha level NA, not in [0,1]
In addition: Warning message:
In grDevices::rgb(col[1, ], col[2, ], col[3, ], alpha) :
NAs introduced by coercion

my sessionInfo() is

R version 3.3.1 (2016-06-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] earth_4.4.7 plotmo_3.2.0 TeachingDemos_2.10 plotrix_3.6-3 classifierplots_1.3.2
[6] data.table_1.10.4 mlr_2.10 ParamHelpers_1.10 reshape_0.8.6 fastICA_1.2-0
[11] corrplot_0.77 e1071_1.6-8 dplyr_0.5.0 purrr_0.2.2 readr_1.0.0
[16] tidyr_0.6.1 tibble_1.2 ggplot2_2.2.1 tidyverse_1.1.1

loaded via a namespace (and not attached):
[1] httr_1.2.1 jsonlite_1.1 splines_3.3.1 foreach_1.4.3 modelr_0.1.0 gtools_3.5.0
[7] LiblineaR_1.94-2 assertthat_0.1 stats4_3.3.1 deepnet_0.2 backports_1.0.3 lattice_0.20-33
[13] quantreg_5.29 checkmate_1.8.2 rvest_0.3.2 minqa_1.2.4 colorspace_1.3-2 Matrix_1.2-6
[19] plyr_1.8.4 psych_1.6.12 broom_0.4.1 SparseM_1.74 haven_1.0.0 caret_6.0-73
[25] corpcor_1.6.8 scales_0.4.1 parallelMap_1.3 gdata_2.17.0 sda_1.3.7 fdrtool_1.2.15
[31] MatrixModels_0.4-1 lme4_1.1-12 mgcv_1.8-12 car_2.1-3 xgboost_0.6-4 pacman_0.4.1
[37] ROCR_1.0-7 nnet_7.3-12 lazyeval_0.2.0 pbkrtest_0.4-6 mnormt_1.5-5 survival_2.39-4
[43] magrittr_1.5 readxl_0.1.1 nlme_3.1-128 MASS_7.3-45 gplots_3.0.1 forcats_0.2.0
[49] xml2_1.1.0 foreign_0.8-66 class_7.3-14 tools_3.3.1 hms_0.3 BBmisc_1.10
[55] stringr_1.1.0 munsell_0.4.3 entropy_1.2.1 caTools_1.17.1 grid_3.3.1 nloptr_1.0.4
[61] iterators_1.0.8 labeling_0.3 bitops_1.0-6 gtable_0.2.0 ModelMetrics_1.1.0 codetools_0.2-14
[67] DBI_0.5-1 reshape2_1.4.2 R6_2.1.3 gridExtra_2.2.1 lubridate_1.6.0 KernSmooth_2.23-15