Unrealistic perplexity about berkeleylm HOT 3 OPEN

Sorry for taking so long to get back. My first guess is that it has something 
to do with the score for unseen words. Can you verify that running scoring the 
data you generated the LM from (so that there are no unknown words) with both 
SRILM and BerkeleyLM gives similar results? Otherwise, it might be some ugly 
character encoding issues. 

It is better but still an order of magnitude smaller (in absolute value)
than that of SRILM. My corpus is encoded in UTF-8. Vietnamese text makes
heavy use of accented characters which cannot be represented in ASCII.


$ . ./env.sh

$ java -ea -mx1000m -server -cp berkeleylm.jar
edu.berkeley.nlp.lm.io.MakeKneserNeyArpaFromText 5 segmented.arpa
$SEGMENTED_CORPUS_TRAIN
$ java -ea -mx1000m -server -cp berkeleylm.jar
edu.berkeley.nlp.lm.io.MakeLmBinaryFromArpa segmented.arpa segmented.binary
$ java -ea -mx1000m -server -cp berkeleylm.jar
edu.berkeley.nlp.lm.io.ComputeLogProbabilityOfTextStream segmented.binary
$SEGMENTED_CORPUS_TRAIN
Log probability of text is: *-67358.47160708543*

$ ngram-count -ukndiscount -order 5 -lm segmented.srilm.arpa -text
$SEGMENTED_CORPUS_TRAIN
$ ngram -lm segmented.srilm.arpa -ppl $SEGMENTED_CORPUS_TRAIN
file segmented.train.txt: 68197 sentences, 1.54738e+06 words, 0 OOVs
0 zeroprobs, logprob= *-3.16751e+06* ppl= 91.3297 ppl1= 111.435

Interesting. Full disclosure: I don't have time to do real debugging anymore 
myself, so I think you're largely on your own. SRILM by default does different 
things with modified KN smoothing and computation of discount factors. At one 
point, I made sure they did exactly the same thing for some simplified settings 
of SRILM, but I couldn't tell you what those settings are. 

If I were you, I check very short sentence with very common words. Most of the 
difference between SRILM and BerkeleyLM happens for low-count words, so the 
difference should shrink if that's all that's going on.