This repository contains the synthetic language generator used in the paper:

"On the Difficulty of Translating Free-Order Case-Marking Languages"
A. Bisazza, A. Ustun, S. Sportel. To appear in TACL (2021).

where we used synthetic languages to study whether free-order case-marking languages (such as Russian, Turkish or Tamil) are more difficult to machine-translate than their fixed-order counterparts (e.g. English). For details and findings on Neural Machine Translation models (specifically, bi-LSTM and Transformers) see the paper.

But what are synthetic languages? The idea is to generate synthetic versions of the English language differing from it in only one or few selected typological features. For instance, the original sentence’s order (SVO) is transformed to different orders, like SOV or VSO, based on its syntactic parse tree. A different kind of variation is the addition of artificial case markers to argument heads.

Our generator is a modified version of the code for the creation of the synthetic languages used in the paper:

"Studying the inductive biases of RNNs with synthetic variations of natural languages"
S. Ravfogel, Y. Goldberg, T. Linzen, NAACL 2019.
https://github.com/shauli-ravfogel/rnn_typology

While Ravfogel et al. (2019) focused on the task of verb-agreement prediction, we adapted the generator for the task of Machine Translation (e.g. modifying the source side of an English-French parallel corpus). We also modified and extended the artificial morphological paradigms to simulate a language with declensions.

The dataset_creation directory contains the code for transforming an existing English dataset into a synthetic version of English.

The arguments specified in main.py allow controlling for various parameters, such as the order of core arguments (sov, svo, ovs, ..., random) and the kind of case marking to be added (if any).

• example input:
  "the woman says her sisters often invited her for dinner"

• examples output:

  • SOV, no case:
    "the woman her sisters her often invited for dinner say"

  • SOV, unambiguous case marking (overt):
    "the woman.nsubj.sg her sisters.nsubj.pl she.dobj.sg often invited.dobj.sg.nsubj.pl for dinner say.nsubj.sg"

  • SOV, unambiguous case marking (implicit with declensions):
    "the womankar her sisterspon shekit often invitedkitpon for dinner saykar"

More examples are provided in Bisazza et al. (2021).

1. Run ./setup.sh to setup a virtual environment and install the required packages
2. Run sbatch jobscript.sh [sov|svo|ovs|osv|vso|...] [na-d na-s na-a] [0|1] [random_seed_int] or jobscript_cases.sh [sov|svo|ovs|osv|vso|...] [na-d na-s na-a] [0|1] [random_seed_int], this will
   • let the code modify the data contained in datasets_creation/data/dev-penn-ud.zip
   • write output data to /data/$USER/rnn_typology/deps_[wo].csv, /data/$USER/rnn_typology/en_[wo][agreement-marker]seed[int].json.gz and /data/$USER/rnn_typology/en_[wo][agreement-marker]seed[int].ftr (Feather file format)

If using this code, please cite the two papers above!