kmcs-nii / planetext Goto Github PK

The program is pretty slow for huge databases. Because of the GIL in ruby, it is not so easy to parallelize the process. On the other hand, if we use JRuby (which can be used in parallel easily) has a very poor and extremely slow implementation of nokogiri (which is necessary to parse the HTML).

So either find a way around GIL in ruby or fix the performance issues in nokogiri in JRuby or found a new workaround.

Goran-san>
I've found that there appears a warning about a possible security issue on the top of this repository. Could you please have a look at it and resolve it?

Best,
Akiko

Per default, planetext put the plain text of annotations to the .ann files. For example, if we define MathML as objects for replacements, the MathML in the .ann files only consists of the UTF-8 characters in the MathML data, e.g.

#3	AnnotatorNotes T3	α 𝛼 italic-alpha \alpha

rather than the MathML data itself.

I currently fixed this issue by manually adding an if-else-case in

It would be better if we can handle this with an extra option in the config yaml. For example, :objectescapexml: or something like that.