loanpydatahub / gothichungarian Goto Github PK

• replace recipdf_idx with ID of reconstruct.csv
• add ID of adapt.csv
• add an ID of the new table itself
• rename Meaning_x and Meaning_y to Meaning_ad and Meaning_rc
• Check if the IDs and Meanings didn't get messed up
• add a write option to sanity.Search.phonmatch, to not having to re-run that part every time (took 13mins now)
• Check how many of the meanings are contained in the word-vectors. This is done with model.has_index_for(tr) (see helpers.gensim_multiword). Try improving this number
• If cleaning the meaning-column doesn't work, consider using BERT or some other sentence-based semantic similarity judgement algorithm
• Check what is more of a reason for failure: semantic or phonological matching
• Double-check subtasks in phonological matching: sound law extraction, application, matching

Exclude words from analysis that are not likely to be old because they appear in texts late

We have a new routine, according to which we now download data with GIT and put entire CLDF directories into raw.

We have used this in a very "professional" way here: https://github.com/lexibank/lexibank-analysed/blob/main/cldfbench_lexibank_analysed.py#L95

But that is a bit too much for you, as you don't need to have all data on Zenodo for development stage.

So you can do something like:

    def cmd_download(self, args):
        for dataset in self.etc_dir.read_csv(
                "datasets.tsv", 
                delimiter="\t",
                dicts=True):
            if self.raw_dir.joinpath(
                    dataset["ID"], "cldf", "cldf-metadata.json").exists():
                args.log.info("skipping {0}".format(dataset["ID"]))
            else:
                args.log.info("cloning {0} to raw/{0}".format(dataset["ID"]))
                repo = Repo.clone_from(
                        "https://github.com/"+
                        dataset["Organisation"]+"/"+
                        dataset["Repository"]+'.git',
                        self.raw_dir / dataset["ID"])

You only need a file etc/datasets.tsv which contains ID, Organisation, and Repository as key, and ID and Repository are typically identical.