Turchin about autocogphylo HOT 11 CLOSED

here's the code to run it:

from lingpy import *
from glob import glob

files = ['ABVD_full.txt', 'IELex-2016.tsv']
for f in files[::-1]:
    print(f)
    lex = LexStat('data/'+f, check=True, transcription='transcription')
    lex.cluster(method='turchin', ref='cogs')
    lex.output('paps.nex', filename=f, ref='cogs', missing='?')

but I just checked: this will yield some errors, both in ielex and abvd, not many, but still, so it may be useful to check that or have it checked.

from autocogphylo.

here's the nexus for IELex:

IELex-2016.tsv.paps.nex.txt

from autocogphylo.

@PhyloStar, tell me, how you prefer to handle this: do you want me to submit simple scripts, which take care of things, so you call them from command line, or rather just point to code solutions and have you do the lifting, so you can see that it is a more integrated framework?

from autocogphylo.

@LinguList Simple scripts might be sufficient. Running LexStat on the whole of ABVD might take sometime.

I am using ASJP sound classes for running LDN and PMI. Do you think we need to stick to SCA or ASJP or test with both?

from autocogphylo.

test with both or only with one should not make a big difference, I'd say...

from autocogphylo.

Okay. Lets say I test LDN and PMI with ASJP. Doing with ASJP is part of tradition. :) Whereas, SCA+LexStat/Turchin is part of tradition also. :)

from autocogphylo.

I get a memory error when running Turchin on the ABVD full dataset.

from autocogphylo.

from autocogphylo.

Here's hou you compute turchin:

dolgos = {}
wl = Wordlist('ABVDxxx.tsv')
for idx, segments in iter_rows(wl, 'tokens'):
    dolgo = tokens2class(tokens, 'dolgo') + ['H']
    if dolgo[0] == 'V':
        dolgo = ['H'] + dolgo
    dstring = ''.join([d for d in dolgo if d != 'V'])[:2]
    dolgos[idx] = dstring+'-'+wl[k, 'concept']
wl.add_entries('cog', dolgos, lambda x: dolgos[x])
wl.renumber('cog', 'turchinid')
wl.output('paps.nex', ref='turchinid', missing='?')

from autocogphylo.

The basic idea is: just convert all segments to the two-letter thing in dolgopolsky-strings, and then use lingpy's functions for renumbering of cognate sets to convert the things to numbers. Much faster and easier than envoking the lexstat model.

from autocogphylo.

jusut added the code "turchin.py", note that this is much faster and closer to the original description

from autocogphylo.