ArrayOutOfBoundsException when reading in a large ARPA file. about berkeleylm HOT 7 CLOSED

Removing all non-alphanumeric characters (save spaces), the data set gets read 
in fine. 

It seems like there are some characters (` maybe?) that it can't handle. Is 
there a list somewhere I can find of unsupported characters? Is this sounding 
right to you?

Can you print out the exact line that's failing? I don't know why it wouldn't 
handle special characters. 

Do you mean the line in my ARPA file? I don't know, that's part of my problem.

Here's the ARPA file that's not working.

On line edu.berkeley.nlp.lm.io.ArpaLmReader.parseLine(ArpaLmReader.java:172), 
add a print statement that prints the |line| if the array is less than the 
number of spaces in |line| is less than |ngram.length|.

Ah, I figured it out. My parser was giving my "words" that still had spaces in 
them, so I would write a unigram that the parser was interpreting as a bigram. 
(I am using stanford's NLP parser to parse files into sentences).

Your hint about spaces helped. Thanks!