The modlm from neubig

Segmentation fault on process.sh

I try to run example but the execution of process.sh results in "Segmentation fault".
The contents of example/result-ptb/mkn+w1x200-lstm200x1-sent-adam-mb0512-0.001_o-1-s100.log are as follows.

/opt/modlm/example/../src/modlm/modlm-train --cnn_seed 100 --cnn_mem 0512 --wildcards "all aa ab ac ad ae af ag ah ai aj" --vocab_file model-ptb/ptb.train.vocab --train_file data-ptb/ptb.held.WILD.txt --valid_file data-ptb/ptb.valid.txt --test_file data-ptb/ptb.test.txt --dist_models "model-ptb/ptb.train.WILD.ngram_mkn_4_3_2_1 model-ptb/ptb.train.all.uniform model-ptb/ptb.train.all.unk" --learning_rate 0.001 --rate_decay 1.0 --model_dropout_prob 0.0 --model_dropout_decay 1.0 --dropout_models "" --trainer adam --training_type sent --clipping_enabled true --layers lstm:200:1 --node_dropout 0.0 --word_hist 1 --word_rep 200 --penalize_unk false --epochs 1000 --online_epochs -1 --max_minibatch 0512 --whiten "" --whiten_eps 0.001 --heuristic "" --evaluate_frequency 1 --model_out result-ptb/mkn+w1x200-lstm200x1-sent-adam-mb0512-0.001_o-1-s100.mod 2>&1 >> result-ptb/mkn+w1x200-lstm200x1-sent-adam-mb0512-0.001_o-1-s100.log &
[cnn] random seed: 100
[cnn] allocating memory: 0512MB
[cnn] memory allocation done.
Started training! (s=7.51e-07)
Reading vocabulary... (s=8.7075e-05)
Started reading model model-ptb/ptb.train.all.ngram_mkn_4_3_2_1 (s=0.00298226)
Started reading model model-ptb/ptb.train.all.uniform (s=5.57657)
Started reading model model-ptb/ptb.train.all.unk (s=5.57661)
Finished reading models (s=5.57662)
Creating model (s=5.57663)
Creating data for data-ptb/ptb.valid.txt (s=5.57856)
Creating data for data-ptb/ptb.test.txt (s=5.82293)
Started reading model model-ptb/ptb.train.aa.ngram_mkn_4_3_2_1 (s=6.09834)
Creating data for data-ptb/ptb.held.aa.txt (s=11.2268)
Started reading model model-ptb/ptb.train.ab.ngram_mkn_4_3_2_1 (s=11.8082)
Creating data for data-ptb/ptb.held.ab.txt (s=16.8796)
Started reading model model-ptb/ptb.train.ac.ngram_mkn_4_3_2_1 (s=17.4329)
Creating data for data-ptb/ptb.held.ac.txt (s=22.7245)
Started reading model model-ptb/ptb.train.ad.ngram_mkn_4_3_2_1 (s=23.0686)
Creating data for data-ptb/ptb.held.ad.txt (s=28.345)
Started reading model model-ptb/ptb.train.ae.ngram_mkn_4_3_2_1 (s=28.6992)
Creating data for data-ptb/ptb.held.ae.txt (s=33.9831)
Started reading model model-ptb/ptb.train.af.ngram_mkn_4_3_2_1 (s=34.308)
Creating data for data-ptb/ptb.held.af.txt (s=39.6078)
Started reading model model-ptb/ptb.train.ag.ngram_mkn_4_3_2_1 (s=39.9576)
Creating data for data-ptb/ptb.held.ag.txt (s=45.2559)
Started reading model model-ptb/ptb.train.ah.ngram_mkn_4_3_2_1 (s=45.5813)
Creating data for data-ptb/ptb.held.ah.txt (s=50.8563)
Started reading model model-ptb/ptb.train.ai.ngram_mkn_4_3_2_1 (s=51.1832)
Creating data for data-ptb/ptb.held.ai.txt (s=56.4738)
Started reading model model-ptb/ptb.train.aj.ngram_mkn_4_3_2_1 (s=56.8594)
Creating data for data-ptb/ptb.held.aj.txt (s=62.1225)
Done creating data. Whitening... (s=62.7943)
--- Starting epoch 1: online, lr=0.001 (s=63.2522)
./process.sh: line 47: 10561 Segmentation fault      (core dumped) $MDIR/src/modlm/modlm-train --cnn_seed $SEED --cnn_mem $MEM --wildcards "all aa ab ac ad ae af ag ah ai aj" --vocab_file model-$SUF/$DATA.vocab --train_file data-$SUF/$HELDW.$LANG --valid_file data-$SUF/$VALID.$LANG --test_file data-$SUF/$TEST.$LANG --dist_models "$MODELS" --learning_rate $LR --rate_decay $RD --model_dropout_prob $DROPOUT_PROB --model_dropout_decay $DROPOUT_DECAY --dropout_models "$DROPOUT_MODELS" --training_type $TRAINING_TYPE --trainer $TRAINER --clipping_enabled $CLIP --layers "$RECURRENCE:$NODES:$LAYERS" --node_dropout $NODE_DROPOUT --word_hist $WHIST --word_rep $WREP --penalize_unk false --epochs $EPOCHS --online_epochs $OEPOCHS --max_minibatch $MINIBATCH --whiten "$WHITEN" --whiten_eps $WHITEN_EPS --heuristic "$HEURISTIC" --evaluate_frequency $EVAL_FREQ --model_out result-$SUF/$LOGID.mod

I'm using dependent libraries of the following revisions.

cnn revision: b39e16e2879df7f8043b192caaaf71b8d58cd78e
Eigen revision: 9546

Thanks!

which branch of dynet to use?

The README says to use the "v2" branch of dynet. As far as I can tell, there is no branch of dynet called "v2". I can't get this repo to build using the master branch of dynet.

Meaning of the penalize_unk flag

I'm trying to figure out how 's are handled during training and evaluation. There is a flag called "penalize_unk" that can be set to true or false. If it is true then I expect s to be treated the same as any word in the vocabulary. If it is false then I expect the 's to be ignored during training and testing.

The first issue is that there is no way to remove unk's from the n-gram models. This means that these will always assign probability mass to the unk token even when we are telling it that there are no unk's in the test data.

The second issue is that when the penalize_unk flag is false, the way the average log likelihood seems to be calculated is to not include in the numerator but to still include it in the denominator. If there are a lot of unks then the perplexity becomes really low. What I would expect is that the unk's should not be counted in both the numerator and the denominator. I'm pretty sure this is what SRILM does. This behavior is non-intuitive and makes me wonder why the penalize_unk flag is set to false in the example script.

I'm still working on understanding all the code. If I can figure it out myself I will try and submit a fix myself.

neubig / modlm Goto Github PK

modlm's People

Contributors

Stargazers

Watchers

Forkers

modlm's Issues

Segmentation fault on process.sh

which branch of dynet to use?

Meaning of the penalize_unk flag

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent