jackstat / modelmetrics Goto Github PK

View Code? Open in Web Editor NEW

29.0 3.0 9.0 527 KB

Rapid Calculation of Model Metrics

R 60.59% C++ 39.41%

metrics auc logloss model-metrics machine-learning model-evaluation

modelmetrics's Introduction

ModelMetrics: Rapid Calculation of Model Metrics

Tyler Hunt [email protected]

Introduction

ModelMetrics is a much faster and reliable package for evaluating models. ModelMetrics is written in using Rcpp making it faster than the other packages used for model metrics.

Installation

You can install this package from CRAN:

install.packages("ModelMetrics")

Or you can install the development version from Github with devtools:

devtools::install_github("JackStat/ModelMetrics")

Benchmark and comparison

N = 100000
Actual = as.numeric(runif(N) > .5)
Predicted = as.numeric(runif(N))

actual = Actual
predicted = Predicted

s1 <- system.time(a1 <- ModelMetrics::auc(Actual, Predicted))
s2 <- system.time(a2 <- Metrics::auc(Actual, Predicted))
# Warning message:
# In n_pos * n_neg : NAs produced by integer overflow
s3 <- system.time(a3 <- pROC::auc(Actual, Predicted))
s4 <- system.time(a4 <- MLmetrics::AUC(Predicted, Actual))
# Warning message:
# In n_pos * n_neg : NAs produced by integer overflow
s5 <- system.time({pp <- ROCR::prediction(Predicted, Actual); a5 <- ROCR::performance(pp, 'auc')})


data.frame(
  package = c("ModelMetrics", "pROC", "ROCR")
  ,Time = c(s1[[3]],s3[[3]],s5[[3]])
)

# MLmetrics and Metrics could not calculate so they are dropped from time comparison
#        package   Time
# 1 ModelMetrics  0.030
# 2         pROC 50.359
# 3         ROCR  0.358

modelmetrics's People

Contributors

Stargazers

Watchers

Forkers

kush99993s decisionpatterns huangmozhilv ashmstat mcuma hack-r tczhao labrax lewinfox

modelmetrics's Issues

Suggestion for improvement of the auc function

I noticed that your version of auc is still about one third slower than the one I recently implemented in mlr. One bottleneck seems to be this line:

if(class(actual) %in% c('factor', 'character')){
   actual = as.numeric(as.factor(as.character(actual))) - 1
 }

I suggest to replace it with

if (inherits(actual, 'factor')) {
    actual <- as.integer(actual) - 1L
  } else if (inherits(actual, 'character')) {
    actual <- as.integer(as.factor(actual)) - 1L
  }

See the following benchmark:

library(ModelMetrics)
library(mlr)
library(microbenchmark)
library(data.table)
x <- c('Pos', 'Neg')
actual <- sample(factor(x, x), 50000, replace = T)
predicted <- runif(length(actual))

auc3_ <- ModelMetrics:::auc3_
binaryChecks <- ModelMetrics:::binaryChecks
auc2 <- function(actual, predicted, ...) {
  binaryChecks(actual, 'auc')
  if (inherits(actual, 'factor')) {
    actual <- as.integer(actual) - 1L
  } else if (inherits(actual, 'character')) {
    actual <- as.integer(as.factor(actual)) - 1L
  }
  
  if(length(actual > 10000)){
    ranks = frankv(predicted)
    AUC <- ModelMetrics:::auc3_(actual, predicted, ranks)
  } else {
    AUC <- auc_(actual, predicted, ranks)
  }
  return(AUC)
}

microbenchmark(mlr = measureAUC(predicted, actual, positive = 'Pos'),
               modelmetrics = auc(actual, predicted),
               modelmetrics.improved = auc2(actual, predicted))

Unit: milliseconds
                  expr      min       lq     mean   median       uq      max neval cld
                   mlr 4.148332 4.236479 4.454990 4.305307 4.445979 6.128918   100 a  
          modelmetrics 6.395471 6.496297 6.840453 6.631536 6.790019 9.250582   100   c
 modelmetrics.improved 4.480996 4.534126 4.836400 4.623480 4.722645 6.802999   100  b

Specificity returns NaN

I got this behavior which looks like a bug. It appears both in the package from CRAN and in the latest version from GitHub. Here is a reproducible example:

library(ModelMetrics)
observed <- structure(c(1L, 1L, 2L, 1L, 1L, 2L, 1L, 1L, 1L, 1L), .Label = c("+", 
"-"), class = "factor")
yprob <- c(0.11, 0.19, 0.89, 0.3, 0.33, 0.92, 0.48, 0.03, 0.09, 0.13)

sensitivity(observed, yprob, 0.1)
sensitivity(observed, yprob, 0.2)
sensitivity(observed, yprob, 0.3)
sensitivity(observed, yprob, 0.4)
sensitivity(observed, yprob, 0.5)

specificity(observed, yprob, 0.6)
specificity(observed, yprob, 0.7)
specificity(observed, yprob, 0.8)
specificity(observed, yprob, 0.9)
specificity(observed, yprob, 1.0)

Notice that sensitivity() accepts a factor as input but the same is not valid for specificity(). Using as.numeric(observed) doesn't work, but it is ok if as.numeric(observed == "+") is used.

Resolve discrepancy between mlogloss and mauc

mlogloss expects a matrix while mauc expects a data.frame. One of them needs to change. mauc is a new function and not in the CRAN release so it may be the one that needs more thought

Compilation syntax issue on package install

While trying to install ModelMetrics from GitHub, I obtain the following compilation error.

auc_.cpp: In function ‘Rcpp::NumericVector avg_rank(Rcpp::NumericVector)’:
auc_.cpp:37:20: error: expected ‘=’ before ‘,’ token
     for (R_xlen_t n, i = 0; i < sz; i += n) {
                    ^
auc_.cpp:37:29: error: ‘i’ was not declared in this scope
     for (R_xlen_t n, i = 0; i < sz; i += n) {
                             ^
make: *** [/usr/lib64/R/etc/Makeconf:166: auc_.o] Error 1

Similar error happens with installation from CRAN as well. My C++ skills could be better but it looks like that taking n definition outside the for loop fixes it, i.e. replacing the above with this seems to compile just fine.

R_xlen_t n;
#pragma omp parallel for
for (int i = 0; i < sz; i += n) {

Has there been a syntax change recently?

System's C++ compiler is g++ (GCC) 7.3.1 20180712 (Red Hat 7.3.1-6). Here's some info on my R version as well.

> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Fedora 27 (Twenty Seven)

Matrix products: default
BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so

locale:
 [1] LC_CTYPE=en_US.utf8       LC_NUMERIC=C             
 [3] LC_TIME=en_US.utf8        LC_COLLATE=en_US.utf8    
 [5] LC_MONETARY=en_US.utf8    LC_MESSAGES=en_US.utf8   
 [7] LC_PAPER=en_US.utf8       LC_NAME=C                
 [9] LC_ADDRESS=C              LC_TELEPHONE=C           
[11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C      

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.5.1

Add vignette with general use cases

add vignette with xgboost examples

accuracy not found

Hi,

It seems that accuracy cannot be found in this package.

Best,

Shixiang

use "average" for ranking ties to get AUC to calculate correctly

Prep for CRAN

Add Vignettes
Review/Revise gini
Version number
Documentation review

error ticket

Testing framework

Gini for continuous response

I see that the package implements Gini for binary response. Nevertheless in many application it could be used also when the response is not negative (insurance losses for example). A five - year old code found in Kaggle is this:

normalizedGini <- function(aa, pp) {
  .Gini <- function(a, p) {
    if (length(a) !=  length(p)) stop("Actual and Predicted need to be equal lengths!")
    temp.df <- data.frame(actual = a, pred = p, range=c(1:length(a)))
    temp.df <- temp.df[order(-temp.df$pred, temp.df$range),]
    population.delta <- 1 / length(a)
    total.losses <- sum(a)
    null.losses <- rep(population.delta, length(a)) # Hopefully is similar to accumulatedPopulationPercentageSum
    accum.losses <- temp.df$actual / total.losses   # Hopefully is similar to accumulatedLossPercentageSum
    gini.sum <- cumsum(accum.losses - null.losses)  # Not sure if this is having the same effect or not
    sum(gini.sum) / length(a)
  }
  .Gini(aa,pp) / .Gini(aa,aa)
}

Would be possible to enrich the function available in the package with with feature? Best

Gini for continuous responses

I was trying to use the Gini function with a non negative score but I receive the following Error:

out<-with(db4scoring, ModelMetrics::gini(actual=numero, predicted = numeroPred))
Error in binaryChecks(actual, "auc") :
auc only works for binary outcomes at this time

Area Under Precision Recall Curve

Fix F1 score

The F1 score is not even close...

imminent CRAN archiving of ModelMetrics

On January 11, 2020 at 4:12:35 AM, Kurt Hornik ([email protected]) wrote:

Dear maintainers,

This concerns the CRAN packages

ModelMetrics caret vip

maintained by one of you:

Brandon Greenwell [email protected]: vip
Max Kuhn [email protected]: caret
Tyler Hunt [email protected]: ModelMetrics

We have repeatedly asked for an update fixing the check problems
shown on
https://cran.r-project.org/web/checks/check_results_ModelMetrics.html
with no reply from the maintainer thus far.

Thus, package ModelMetrics is now scheduled for archival on 2020-01-25,
and archiving this will necessitate also archiving its strong reverse
dependencies.

Please negotiate the necessary actions.

Best
-k

Add function for multiclass auc

should be easy to run normal auc on each outcome and then average them

Option to add names to the output of confusionMatrix()

Hi, thanks for this super helpful package!

I often get confused (no pun intended) as to which axis of the confusion matrix is which, particularly because some packages (e.g h2o) use different conventions.

How would you feel about adding an option to confusionMatrix() to add dimnames to the result?

actual <- c(0, 0, 1)
predicted <- c(1, 0, 1)

# The current default behaviour is unchanged
confusionMatrix(actual, predicted)
#>      [,1] [,2]
#> [1,]    1    0
#> [2,]    1    1

# But we can add labels if desired
confusionMatrix(actual, predicted, use_names = TRUE)
#>             actual_0 actual_1
#> predicted_0        1        0
#> predicted_1        1        1

If you think this has value I have an implementation here and can submit a PR for review.

Thanks!

R2 metric

Hi @JackStat is it possible to add the $R^2$ metrics to the package for numeric responses? $R^2=\frac{explained variance}{total variance}=\frac{\sum\left( \hat{y}-\bar{y}\right)^2}{\sum\left( y-\bar{y}\right)^2}$

Add function for Neyman-Pearson receiver operating characteristic

http://arxiv.org/pdf/1608.03109v1.pdf

Also a version written in R in the rproc package

new mlogLoss warnings in r-devel

mlogLoss issues warnings in R4.0 from this line:

class(predicted) %in% c("data.frame")

since matrices have extra classes now. You might want to wrap this in any() or use inherits(predicted, "data.frame") or is.data.frame().

Examples in readme

Badges would be good too
examples in readme
benchmarks and package comparisons

Add function for multi-class logloss

Continuous integration

handle missing values

segfault in R3.4.3 on Mac

@JackStat FYI Running caret with metric "ROC" consistently produces a segfault on macOS (sierra) with R3.4.3. The segfault message is shown below. Example code also shown below. I have been able to reproduce on multiple different datasets and estimation algorithms. The segfault message refers to ModelMetrics_auc. The code example works fine if I provide my own function to calculate auc.

I hope you'll be able to take a look at this. Please let me know if you need more information. Thanks

cc-ing @topepo

 *** caught segfault ***
address 0x18, cause 'memory not mapped'

Traceback:
 1: .Call("ModelMetrics_auc_", PACKAGE = "ModelMetrics", actual,     predicted, ranks)
 2: auc_(actual, predicted, ranks)
 3: ModelMetrics::auc(ifelse(data$obs == lev[2], 0, 1), data[, lvls[1]])
 4: ctrl$summaryFunction(testOutput, lev, method)
 5: evalSummaryFunction(y, wts = weights, ctrl = trControl, lev = classLevels,     metric = metric, method = method)
 6: train.default(x, y, weights = w, ...)
 7: train(x, y, weights = w, ...)
 8: train.formula(vs ~ ., data = dat, method = "ranger", trControl = ctrl,     tuneGrid = grid, metric = "ROC", verbose = FALSE)
 9: train(vs ~ ., data = dat, method = "ranger", trControl = ctrl,     tuneGrid = grid, metric = "ROC", verbose = FALSE)

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 
Selection: 
Selection:

## loading libraries
library(ranger)
library(caret)

dat <- mtcars
dat$vs <- factor(ifelse(dat$vs == 1, "yes", "no"))
sapply(dat, class)

ranger(
  vs ~ ., 
  data = dat, 
  probability = TRUE, 
  num.trees = 50, 
  mtry = 3
)

set.seed(1234)
grid <- expand.grid(mtry = 3:4, splitrule = "gini", min.node.size = 1)
ctrl <- trainControl(
  method = "cv", 
  number = 5, 
  classProbs = TRUE, 
  summaryFunction = twoClassSummary, 
  verboseIter = TRUE
)

result <- train(
  vs ~ ., 
  data = dat,
  method = "ranger", 
  trControl = ctrl, 
  tuneGrid = grid, 
  metric = "ROC", 
  verbose = FALSE
)

Session Info:

> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] caret_6.0-78    ggplot2_2.2.1   lattice_0.20-35 ranger_0.9.0   

loaded via a namespace (and not attached):
 [1] tidyselect_0.2.3   purrr_0.2.4        reshape2_1.4.3     kernlab_0.9-25     splines_3.4.3      colorspace_1.3-2  
 [7] stats4_3.4.3       yaml_2.1.16        survival_2.41-3    prodlim_1.6.1      rlang_0.2.0.9000   ModelMetrics_1.1.0
[13] pillar_1.1.0       withr_2.1.1        foreign_0.8-69     glue_1.2.0         bindrcpp_0.2       foreach_1.4.3     
[19] bindr_0.1.0.9000   plyr_1.8.4         dimRed_0.1.0       lava_1.6           robustbase_0.92-8  stringr_1.3.0     
[25] timeDate_3042.101  munsell_0.4.3      gtable_0.2.0       recipes_0.1.2      codetools_0.2-15   psych_1.7.8       
[31] parallel_3.4.3     class_7.3-14       DEoptimR_1.0-8     broom_0.4.3        Rcpp_0.12.15       scales_0.5.0      
[37] ipred_0.9-6        CVST_0.2-1         mnormt_1.5-5       stringi_1.1.6      dplyr_0.7.4        RcppRoll_0.2.2    
[43] ddalpha_1.3.1.1    grid_3.4.3         tools_3.4.3        magrittr_1.5       lazyeval_0.2.1     tibble_1.4.2      
[49] tidyr_0.8.0        DRR_0.0.3          pkgconfig_2.0.1    MASS_7.3-48        Matrix_1.2-12      lubridate_1.7.2   
[55] gower_0.1.2        assertthat_0.2.0   iterators_1.0.8    R6_2.2.2           rpart_4.1-12       sfsmisc_1.1-0     
[61] nnet_7.3-12        nlme_3.1-131       compiler_3.4.3

rmsle returns rmse

rmsle returns sqrt(mse) instead of sqrt(msle).

poisson log loss

The following function could be a nice add for the package, since it implements the Poisson Log Loss in Rcpp (also thanks to SO ) and originally written in R in the MLmetrics package:

#include <Rcpp.h>
#include <math.h>
using namespace Rcpp;

// [[Rcpp::export]]
double poissonLogLoss(NumericVector predicted, NumericVector actual) {
  NumericVector temp, y_pred_new;
  double out; 
  const double eps=1e-15;

  y_pred_new=pmax(predicted,eps);
  temp = log(gamma(actual + 1)) + y_pred_new - log(y_pred_new)*actual;
  out=mean(temp); // using sugar implementation
  return out;
}

Failed installation with install.packages('ModelMetrics')

I'm trying to use caret and ModelMetrics. Both installation options fail with:

auc_.cpp:2:10: fatal error: 'omp.h' file not found
#include <omp.h>
         ^~~~~~~
1 error generated.
make: *** [auc_.o] Error 1
ERROR: compilation failed for package 'ModelMetrics'
* removing '/usr/local/lib/R/3.5/site-library/ModelMetrics'
Installation failed: Command failed (1)

I'm on OS X and R Studio 1.1.456

Matthews correlation coefficient

https://en.wikipedia.org/wiki/Matthews_correlation_coefficient

variable results of ModelMetrics::auc on r-devel

When running the code on R4.0, the results are different each time:

library(ModelMetrics)
#> 
#> Attaching package: 'ModelMetrics'
#> The following object is masked from 'package:base':
#> 
#>     kappa
set.seed(52)
N = 100000
Actual = as.numeric(runif(N) > .5)
Predicted = as.numeric(runif(N))

actual = Actual
predicted = Predicted

ModelMetrics::auc(Actual, Predicted)
#> [1] -0.4122159

ModelMetrics::auc(Actual, Predicted)
#> [1] -0.4290166

ModelMetrics::auc(Actual, Predicted)
#> [1] -0.2943601

^{Created on 2019-12-31 by the reprex package (v0.3.0)}

Session info

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                                             
#>  version  R Under development (unstable) (2019-12-29 r77627)
#>  os       macOS Catalina 10.15.1                            
#>  system   x86_64, darwin15.6.0                              
#>  ui       X11                                               
#>  language (EN)                                              
#>  collate  en_US.UTF-8                                       
#>  ctype    en_US.UTF-8                                       
#>  tz       America/New_York                                  
#>  date     2019-12-31                                        
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version    date       lib source                         
#>  assertthat     0.2.1      2019-03-21 [1] CRAN (R 4.0.0)                 
#>  backports      1.1.5      2019-10-02 [1] CRAN (R 4.0.0)                 
#>  callr          3.4.0      2019-12-09 [1] CRAN (R 4.0.0)                 
#>  cli            2.0.0      2019-12-09 [1] CRAN (R 4.0.0)                 
#>  crayon         1.3.4      2017-09-16 [1] CRAN (R 4.0.0)                 
#>  data.table     1.12.8     2019-12-09 [1] CRAN (R 4.0.0)                 
#>  desc           1.2.0      2018-05-01 [1] CRAN (R 4.0.0)                 
#>  devtools       2.2.2      2019-12-31 [1] Github (r-lib/devtools@03537c6)
#>  digest         0.6.23     2019-11-23 [1] CRAN (R 4.0.0)                 
#>  ellipsis       0.3.0      2019-09-20 [1] CRAN (R 4.0.0)                 
#>  evaluate       0.14       2019-05-28 [1] CRAN (R 4.0.0)                 
#>  fansi          0.4.0      2018-10-05 [1] CRAN (R 4.0.0)                 
#>  fs             1.3.1      2019-05-06 [1] CRAN (R 4.0.0)                 
#>  glue           1.3.1      2019-03-12 [1] CRAN (R 4.0.0)                 
#>  highr          0.8        2019-03-20 [1] CRAN (R 4.0.0)                 
#>  htmltools      0.4.0      2019-10-04 [1] CRAN (R 4.0.0)                 
#>  knitr          1.26       2019-11-12 [1] CRAN (R 4.0.0)                 
#>  magrittr       1.5        2014-11-22 [1] CRAN (R 4.0.0)                 
#>  memoise        1.1.0      2017-04-21 [1] CRAN (R 4.0.0)                 
#>  ModelMetrics * 1.2.2      2018-11-03 [1] CRAN (R 4.0.0)                 
#>  pkgbuild       1.0.6      2019-10-09 [1] CRAN (R 4.0.0)                 
#>  pkgload        1.0.2      2018-10-29 [1] CRAN (R 4.0.0)                 
#>  prettyunits    1.0.2      2015-07-13 [1] CRAN (R 4.0.0)                 
#>  processx       3.4.1      2019-07-18 [1] CRAN (R 4.0.0)                 
#>  ps             1.3.0      2018-12-21 [1] CRAN (R 4.0.0)                 
#>  R6             2.4.1      2019-11-12 [1] CRAN (R 4.0.0)                 
#>  Rcpp           1.0.3      2019-11-08 [1] CRAN (R 4.0.0)                 
#>  remotes        2.1.0      2019-06-24 [1] CRAN (R 4.0.0)                 
#>  rlang          0.4.2      2019-11-23 [1] CRAN (R 4.0.0)                 
#>  rmarkdown      2.0        2019-12-12 [1] CRAN (R 4.0.0)                 
#>  rprojroot      1.3-2      2018-01-03 [1] CRAN (R 4.0.0)                 
#>  sessioninfo    1.1.1      2018-11-05 [1] CRAN (R 4.0.0)                 
#>  stringi        1.4.3      2019-03-12 [1] CRAN (R 4.0.0)                 
#>  stringr        1.4.0      2019-02-10 [1] CRAN (R 4.0.0)                 
#>  testthat       2.3.1      2019-12-01 [1] CRAN (R 4.0.0)                 
#>  usethis        1.5.1.9000 2019-12-31 [1] Github (r-lib/usethis@b2e894e) 
#>  withr          2.1.2      2018-03-15 [1] CRAN (R 4.0.0)                 
#>  xfun           0.11       2019-11-12 [1] CRAN (R 4.0.0)                 
#>  yaml           2.2.0      2018-07-25 [1] CRAN (R 4.0.0)                 
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

on R 3.6.1:

library(ModelMetrics)
#> 
#> Attaching package: 'ModelMetrics'
#> The following object is masked from 'package:base':
#> 
#>     kappa
set.seed(52)
N = 100000
Actual = as.numeric(runif(N) > .5)
Predicted = as.numeric(runif(N))

actual = Actual
predicted = Predicted

ModelMetrics::auc(Actual, Predicted)
#> [1] 0.5030778

ModelMetrics::auc(Actual, Predicted)
#> [1] 0.5030778

ModelMetrics::auc(Actual, Predicted)
#> [1] 0.5030778