training finder with xml documents about anystyle HOT 10 OPEN

You can train the parser module with XML files (see the samples in res/parser). However, the finder module is trained using TTX files (see res/finder). TTX is a custom format that makes annotating long text documents relatively easy. Using XML for this would be cleaner but would require a lot of extra tooling to make it feasible (for training documents you need to label every line of a document after all).

Please note that the documents in res/finder are only one part of the sources used to train the default finder module. We can't publish the other documents due to copyrights.

from anystyle.

thank you. So the xml must have that specific structure or i can use my own annotation?
And also, can i use input pdf documents with the parse command or should i provide txt files as input instead?

from anystyle.

The XML used for the parser must use the structure as in the sample files, i.e. one <dataset> with any number of <sequence> records. The tag names used inside a sequence correspond to the labels known to the model, that means you can introduce your own labels there. However, I'd definitely stick with the labels of the default model (author, date, title, etc.) because the feature extraction and normalization helpers operate on those specifically. If you introduce new labels you might want to add your own normalizer code for example to process such segments.

The parse command takes text input (one reference per line). But of course you can use finder and parser module in combination, for example with the CLI tool. The finder module would extract the references from a PDF or text document and pass it on to the parser module which would then segment and label each reference individually.

from anystyle.

Basically, the finder module takes entire documents; it splits the document into lines and operates on each line: every line is assigned a label; multiple lines with the same label are grouped together; reference groups are extracted; a heuristic based on regular expressions is applied to try and separate individual references.

The parser module takes one or more lines as input; each line is interpreted as a single reference; the line is split into word-tokens and each word is labeled; successive words with the same label are grouped together; normalizer routines are applied for specific labels.

from anystyle.

thank you again.Does anystyle provide a converter for creating TTX files?

from anystyle.

Yes you can save documents as TTX. TTX is just plain text but with a certain prefix on each line; it was build for manual annotation using diff and simple text editors like nvim. You can also find more background info in some issue threads here.

from anystyle.

So if i understood correctly:
I could train the parse model by annotating my xml files according to your model and use the trained parser as default when running the find command. Is it correct?
How can I set the trained parser as default?

from anystyle.

When using the CLI tool you can pass the model file as an argument from the command line. If you use the Ruby Gem you can set model option of a given parser instance (or change the default setting).

from anystyle.

while trying to training the parser i got this error
error: undefined method 'strip' for nil:NilClass
but i cannot understand how to fix this. What could it be?

from anystyle.

I think this is probably a cryptic error message due to invalid training data. It's usually something like a blank segment (i.e., something like <title></title>).

from anystyle.