what is the output of compare_fits and what does it mean? about admixtools HOT 7 OPEN

I added some documentation to the function. p_emp is the important item here, the other outputs can mostly be ignored. The function is simply a bootstrap test for testing whether the median score difference is different from zero. It's a bit awkward because the scores are log-likelihoods, but for this test all that matters is that the input vectors come from two models evaluated on bootstrap resampled SNP blocks (using the same resamplings for both models). You could also supply worst f-statistic residuals, or any other estimated model parameter as input.

p_emp is two times the fraction of bootstrap replicates where model 1 has a lower score than model 2 (or vice-vera, whichever is less), truncated at 1/(number of bootstrap replicates).

ci_low and ci_high are the 2.5% and 97.5% quantiles of the score difference distribution. And since the input vectors represent bootstrap replicates, that should be the same as the 2.5% and 97.5% confidence intervals of the score difference distribution.

from admixtools.

from admixtools.

Yes, you can interpret it like a p-value, but it's a two-sided test, so it tests whether either of the two models is significantly better than the other model.

from admixtools.

from admixtools.

Sorry for not being clear!

Low values of p_emp suggest that the difference in scores between the two models is unlikely to be due to the variability across SNP blocks. But this value will be low if model 1 has higher scores than model 2, or the other way round.

If you want to know which of the two models is better, you can look at whether the diff value is positive or negative. That's the estimate of the mean difference scores1 - scores2, so positive means model 1 has higher scores (is worse).

That difference will almost always have the same sign as the score difference that you get when evaluating both models the standard way (using all SNPs with qpgraph, not using bootstrapping with qpgraph_resample_multi). If you have already evaluated both models the standard way, you probably already know which of the two models has a lower score, and you just want to test whether that difference is significant. That's why I didn't emphasize the diff output here!

from admixtools.

from admixtools.

Thanks for the question and answer, but I may need some clarification for some of my results. One of my model (Nadmix = 1) has a score of 0.438 and the other (Nadmix = 2) has a score of 0.000603. However, when I run the qpgraph_resample_multi and copmare_fits functions, I get the following:

$diff
[1] -3.223325
$se
[1,] 6.29504
$z
[1,] -0.5120419
$p
[1] 0.6086217
$p_emp
[1] 0.6
$p_emp_nocorr
[1] 0.6
$ci_low
[1] -22.78158
$ci_high
[1] 2.032134

This suggests to me that none of the models is better than the other. note that I also get a similar result (i.e. no significant p. val.) when comparing these models with higher admixture events (up to 7).

Is my interpretation correct?

Thanks in advance!

from admixtools.