LexStat about autocogphylo HOT 11 CLOSED

Let me check the coverage first ;-)

from autocogphylo.

and reg. the Bouchard-Cote data: it is not really clean, but I'll see.

from autocogphylo.

If possible, please, do not run your modified lexStat code (which might correct for coverage issues). Better to show that LexStat designed for ideal situation can be used to infer decent tree topology.

from autocogphylo.

Okay, ABVD-400 is not a good test set, as the average coverage of the languages is only 53 words!

To replicate, just download most recent lingpy version (from github), last PR, and do (just submitted):

from lingpy.compare.sanity import mutual_coverage
from lingpy import *
from itertools import combinations

wl = Wordlist('data/ABVD_full.txt')
coverage = mutual_coverage(wl)

coperlan = []
for l in wl.cols:
    cov = sum(list(coverage[l].values())) / (wl.width-1)
    print(l, cov)
    coperlan += [cov]
print(sum(coperlan) / len(coperlan))

from autocogphylo.

We could use this code as a base-check for all kind of data.

from autocogphylo.

Okay. Kewl. I still wonder how the trees from ABVD come close to the gold standard tree in the paper of Greenhill et al. (Bayesian myths for Austronesian).

from autocogphylo.

well, I just won't run the code on data with coverage below 100. I think this is a solid way to state it, right?

from autocogphylo.

Okay. Then, I will just exclude those languages from the nexus files for PMI and LDN.

from autocogphylo.

I will be off-grid for two days since I am traveling to India tomorrow. Will be adding pmi nexus files on the way.

Then, I will start running Bayesian analysis for ABVD full and IELex starting with LDN, Turchin, PMI, LexStat.

from autocogphylo.

I added the computed lexstat-scorers in an extra folder which is called bins/. As a default, calling python lexstat.py will take the data from there, use the pre-computed scoring functions, and run the analysis, which is considerably fast. It is a big large in terms of data (130MB), but it's the easiest way to make results comparable, and to speed up re-computation of lexstat scores (also for testing).

from autocogphylo.

I'll close this now, as the data is computed, and lexstat scores are available in scores.md

from autocogphylo.