Calculation of Concordance Correlation Coefficient (CCC) may be incorrect about competitions HOT 2 CLOSED

Hi @JinchaoLove, thanks so much for your comment!

Hi, @aliceebaird, thanks for the kind reply. You're right. And in my given example x = torch.rand(16, 10) # (bsz, n_classes), the CCC should be calculated for each label separately, or we should specify the dimension of the mean function.

from competitions.

Hi @JinchaoLove, thanks so much for your comment!

We have looked into this, and in our implementation, we follow the original implementation from Lin, 1989. This does not affect the training of the baseline model, and the results are not substantially affected by either the biased or unbiased covariance estimation.

Further, as the number of evaluated samples increases, the difference becomes even more negligible. Hence, we do not see this as an issue in our case (train, validation, test: ~15k samples). e.g.,

ns = [100,1000,1000,10000,100000,1000000]
xs = [torch.rand(1, ns[0]) ,torch.rand(1, ns[1]) ,torch.rand(1, ns[2]) , torch.rand(1, ns[3]), torch.rand(1, ns[4]) ]
ys = [xs[0],xs[1],xs[2],xs[3],xs[4]]

for n, x, y in zip(ns,xs,ys):
    print(N: {n}')
    print(f'\t CCC Ours: {CCC(x[0],y[0])}') 
    print(f'\t CCC Other: {CCC_other(x[0],y[0])}')
    
N: 100
	 CCC Ours: 0.9900000035
	 CCC Other: 1.0
...
N: 100000
	 CCC Ours: 0.9999900015
	 CCC Other: 1.0

I will close this issue for now but thank you again, and let us know if you have any other questions or concerns.

from competitions.