posthoc_conover_friedman calculates incorrect p-values about scikit-posthocs HOT 9 CLOSED

Thank you. I have compared the codes for Conover's test in PMCMR and PMCMRplus, and I have noticed that the calculations for T2 are indeed different in the two codes. The correct formula, i.e. consistent with the references, is present in PMCMR. I have already reported this discrepancy in PMCMRplus.

from scikit-posthocs.

Hello! Thank you! Let's try to investigate this one deeper. Could you please run the following R script and post your results here?

library(PMCMRplus)

y <- matrix(c(
  3.88, 5.64, 5.76, 4.25, 5.91, 4.33, 30.58, 30.14, 16.92,
  23.19, 26.74, 10.91, 25.24, 33.52, 25.45, 18.85, 20.45,
  26.67, 4.44, 7.94, 4.04, 4.4, 4.23, 4.36, 29.41, 30.72,
  32.92, 28.23, 23.35, 12, 38.87, 33.12, 39.15, 28.06, 38.23,
  26.65), nrow=6, ncol=6,
  dimnames=list(1:6, LETTERS[1:6])
)

frdAllPairsConoverTest(y, p.adjust.method = 'none')

Here are my results (equal to scikit-posthocs output):

  A      B      C      D      E     
B 0.0367 -      -      -      -     
C 0.0193 0.7710 -      -      -     
D 0.7710 0.0673 0.0367 -      -     
E 0.0098 0.5615 0.7710 0.0193 -     
F 0.0011 0.1537 0.2503 0.0024 0.3858

P value adjustment method: none

from scikit-posthocs.

Also, if you look at PMCMRplus code for frdAllPairsConoverTest, you will find out that scikit-posthocs implementation is very close to it, almost equal in some lines.

I assume that you are using the latest version of PMCMRplus.

from scikit-posthocs.

Hello Max! I'm not familiar with R and don't have the capability to run R scripts. However, I've manually verified the results by substituting values into the formulas as outlined by Conover in his book "Practical nonparametric Statistics" (3rd. Edition, Wiley, 1999; pages 369-371). I suspect parameter T2 might be critical in this regard. PMCMRplus code references Conover's original paper from 1978 which specifies T2 as follows:

However, in PMCMRplus code (and also in scikit-posthocs), T2 is calculated as follows:

PMCMRplus:

  S2 <- m / (m * k - 1) * (sum(r ^ 2) - m * k * n *
                                 (m * k + 1) ^ 2 / 4)
  T2 <- 1 / S2 * (sum(R.sum) - n * m * ((m * k + 1) / 2) ^ 2)

scikit-posthocs:

  R = x.groupby(_group_col)['mat'].sum()
  S2 = m/(m*k - 1.) * (A1 - m*k*n*(m*k + 1.)**2./4.)
  T2 = 1. / S2 * (np.sum(R) - n * m * ((m * k + 1.) / 2.)**2.)

Indeed, the implementation is the same. In both codes, however, the second power is applied only to the parentheses ((m * k + 1) / 2) and is excluded from the summation while Conover applies the second power to the whole term inside the summation (which includes R and the terms with n,m,k). This is not equivalent.

My results corresponded to Pohlert's documentation for PMCMR as shown in my bug report. Maybe Code has changed in PMCMRplus (I didn't check). If so, a bug may have occured with the change.

I compared both Conover's publications (1979 and 1999) and although he gives slightly different formulas in those two publications, they give the same results.

from scikit-posthocs.

Interesting! Thank you! I need some time to look into it. If you have time, please report this in PMCMRplus repo as well

from scikit-posthocs.

Thank you!

from scikit-posthocs.

Here is an email response from the author & maintainer of the PMCMRplus package confirming the bug:

I can confirm the issue and I have already discovered the bug.
I will fix it as soon as possible and will submit it to CRAN.
The results from the older PMCMR package were correct.

from scikit-posthocs.

It seems that the bug has already been fixed in PMCMRplus version 1.9.10. See the news below. Please note that the rdrr snippet for online running the R Code uses version 1.7.0 of the PMCMRplus package and, therefore, contains the bug.

from scikit-posthocs.

Thank you! Fixed in the latest release.

from scikit-posthocs.