This is a super helpful package, however there is a small inconsistency with outcome_b

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Inconsistency in outcome_base about fairness HOT 5 CLOSED

kozodoi commented on May 27, 2024

Inconsistency in outcome_base

from fairness.

Comments (5)

kozodoi commented on May 27, 2024

Hi @urmilaR, thanks a lot for pointing out the issue! I will look into that and will aim to release a fix for prop_parity() in the next fairness 1.2.1 release.

Best,
Nikita

from fairness.

urmilaR commented on May 27, 2024

Hi, is there an update on this?

from fairness.

kozodoi commented on May 27, 2024

Hi @urmilaR, I looked into that but did not have the chance to implement the fix yet. You should expect an update to the GitHub development version by tomorrow evening, and the CRAN release will follow next week. I will let you know as soon as there is an update!

from fairness.

kozodoi commented on May 27, 2024

I corrected the inconsistency issue in fairness 1.2.1 development version on GitHub.

The argument outcome_base was intended to specify a base level of the outcome variable, i.e., a negative class. This means that in your example, equal_odds() was computed incorrectly, whereas prop_parity() was computed correctly. Now this inconsistency is fixed and the documentation for outcome_base is updated to better clarify its meaning. The calculations for all metrics should now be correct.

Following your example snippet above:

Confusion matrix:

caret::confusionMatrix(pred, ref, positive = "1")
Confusion Matrix and Statistics

          Reference
Prediction   0   1
         0 997 434
         1 284 388
                                          
               Accuracy : 0.6586          
                 95% CI : (0.6379, 0.6789)
    No Information Rate : 0.6091          
    P-Value [Acc > NIR] : 1.557e-06       
                                          
                  Kappa : 0.2588          
                                          
 Mcnemar's Test P-Value : 2.688e-08       
                                          
            Sensitivity : 0.4720          
            Specificity : 0.7783          
         Pos Pred Value : 0.5774          
         Neg Pred Value : 0.6967          
             Prevalence : 0.3909          
         Detection Rate : 0.1845          
   Detection Prevalence : 0.3195          
      Balanced Accuracy : 0.6252          
                                          
       'Positive' Class : 1

Calculating equalized odds with outcome_base = '0' (i.e., negative class = 0, positive class = 1):

equal_odds(data         = compas,
           outcome      = 'Two_yr_Recidivism_01',
           probs        = 'probability',
           group        = 'ethnicity',
           cutoff       = 0.5,
           base         = 'Caucasian',
           outcome_base = '0')$Metric

                  Caucasian African_American      Asian    Hispanic Native_American       Other
Sensitivity       0.4720195        0.7525587  0.2500000   0.4656085        0.600000   0.4193548
Equalized odds    1.0000000        1.5943383  0.5296392   0.9864179        1.271134   0.8884270
Group size     2103.0000000     3175.0000000 31.0000000 509.0000000       11.000000 343.0000000

Sensitivity equals 0.4720195, which corresponds to the confusion matrix value.

Calculating proportional parity with outcome_base = '0' (i.e., negative class = 0, positive class = 1):

prop_parity(data         = compas,
            outcome      = 'Two_yr_Recidivism_01',
            probs        = 'probability',
            group        = 'ethnicity',
            cutoff       = 0.5,
            base         = 'Caucasian',
            outcome_base = '0')$Metric
                       Caucasian African_American      Asian    Hispanic Native_American       Other
Proportion             0.3195435         0.591811  0.1290323   0.2927308       0.4545455   0.2478134
Proportional Parity    1.0000000         1.852051  0.4038018   0.9160907       1.4224838   0.7755232
Group size          2103.0000000      3175.000000 31.0000000 509.0000000      11.0000000 343.0000000

Proportion equals 0.3195435, which corresponds to (TP + FP) / (TP + FP + TN + FN) = (284 + 388) / (284 + 388 + 997 + 434) based on the confusion matrix.

You can install the development version using the snippet below and check if it works correctly. I will be doing some further checks in the next 1-2 days as well before submitting the new version to CRAN. Please let me know if you still see any issues.

devtools::install_github('kozodoi/fairness')
library(fairness)

Hope this helps!

Best wishes,
Nikita

from fairness.

kozodoi commented on May 27, 2024

Fairness 1.2.1 with the above fix is now also available on CRAN. Closing the issue.

from fairness.

Inconsistency in outcome_base about fairness HOT 5 CLOSED

Comments (5)

Related Issues (8)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent