a basic probabilistic parser for French based on the CYK algorithm and the PCFG model.
For running successfully this system, you need python 2.7.
Open the terminal, run the command below:
sh run_parser.sh [file] [train_num] [test_num]
- [file]: The input file: some of lines will be used for training, some will be used for test.
- [train_num]: the first several lines in [file] used for training(learning PCFG probas).
- [test_num]: the last several lines in [file] used for test.
sh run_parser.sh ./data/sequoia-corpus+fct.mrg_strict 3098 5
This will take 1-3098 lines in the file "./data/sequoia-corpus+fct.mrg_strict"(totally 3099 lines) as train file, and take 3095-3099 lines in the file as test file.
#!/bin/sh
FILEDIR=$1
PRODDIR="./prod/productions.txt"
TESTFILE="./data/test.txt"
TRAINFILE="./data/train.trees"
GOLDFILE="./data/test.trees"
PARSEFILE="./results/test_parsed.trees"
python2 ./commands/system.py $FILEDIR $TRAINFILE $TESTFILE $GOLDFILE $2 $3
python2 ./commands/pcfg_learn.py $TRAINFILE $PRODDIR
python2 ./commands/cykslover.py $PRODDIR $TESTFILE $PARSEFILE
python2 ./commands/evals.py $PARSEFILE $GOLDFILE
- FILEDIR: The input file path
- PRODDIR: The file storing the result of PCFG
- TESTFILE: The file storing the sentences of test lines
- TRAINFILE: The file storing the parsed tree of train lines
- GOLDFILE: The file storing the parsed tree of test lines
- PARSEFILE: The file storing the parsed tree of test lines derived from the algorithm