Distances between populations. about admixtools HOT 1 OPEN

This is a bit late, but I think you raised an interesting question.

What you call linearity (Dac <= (Dab + Dbc)) is sometimes called triangle inequality, and you are right in that both f2 and (∑|aj-bj| )/M have that property (regardless of how a, b, and c are related to each other), whereas Fst doesn't have it.

f2 doesn't have the proportionality property as you define it, but it has another property which is perhaps more important, and which (∑|aj-bj| )/M doesn't satisfy: If a, b, and c are related like a -> b -> c or like a <- b -> c then f2(a, c) = f2(a, b) + f2(b, c).

This is because f2 is the variance of allele frequency differences between two populations, and the sum of the variances of two independent random variables (drift between a and b and drift between b and c) is equal to the variance of the sum of both (drift between a and c).

See Ancient Admixture in Human History, section Additivity of F2 along a tree branch for more details.

from admixtools.