Analysis plugin for ElasticSearch providing capability for processing inline annotations in documents.

Inline annotations are considered to be simple semantic informations inlined in source text, that are removed from the indexed text and injected as synonyms at positions of the words they are related to.

This plugin provides analyzer AnnotationAnalyzer as well as filter InlineAnnotationFilter. AnnotationAnalyzer is composed of WhitespaceTokenizer, LowerCaseFilter and InlineAnnotationFilter (with default settings). More sophisticated analyzers (equivalent to StandardAnalyzer or SnowballAnalyzer) can be configure via configuration file elasticsearch.yml or web API.

Let's say we have this documents

"Mozart[artist] was born[lifeEvent] in Salzburg[city;Austria]"

If we parse this with StandardAnalyzer equivalent with annotation analysis added to it we get these tokens - some are omitted due to used StopFilter.

                           | [austria]
[artist] | | [lifeevent] | |  [city]
 mozart  | |    born     | | salzburg

If we use StandardAnalyzer the result would be

mozart | artist | | born | lifeevent | | salzburg | city | austria

This plugin follows conventions for elasticsearch plugins, thus can be installed in a standard manner - see http://www.elasticsearch.org/guide/reference/modules/plugins/

To use those custom analyzers/filters you need to either modify elasticsearch.yml configuration file - see http://www.elasticsearch.org/guide/reference/index-modules/analysis/ or specify index mapping via elasticsearch API.

The following example configuration contains definitions for analyzers based on behaviour of StandardAnalyzer and SnowballAnalyzer.

Please note that standard_annotation and snowball_annotation analyzers use standard tokenizer, which removes all non-alphanumeric characters and thus makes it impossible to process inline annotations marked with [,],; (which are used in default behaviour of InlineAnnotationFilter).

For this purpose we need to use mapping char filter, which remaps those special characters to their equivalent, which will be accepted by standard tokenizer as part of the token.

 index :
     analysis :
         char_filter : 
             annotation_remap : 
                 type : mapping
                 mappings : ["[=>__annotation_start__", "]=>__annotation_end__",";=>__annotation_delimiter__"]
         analyzer :                
             standard_annotation :
                 type : custom
                 tokenizer : standard
                 char_filter : annotation_remap
                 filter : [standard, lowercase, annotation_filter, stop]
             snowball_annotation :
                 type : custom
                 tokenizer : standard
                 char_filter : annotation_remap
                 filter : [standard, lowercase, annotation_filter, stop, snowball]
         filter :
             annotation_filter :
                 type : annotation_filter
                 start : __annotation_start__
                 end : __annotation_end__
                 delimiter : __annotation_delimiter__

To test the analyzer you can query the following http://localhost:9200/test/_analyze?analyzer=annotation&text="Mozart[city;Salzburg]"

Another thing to keep in mind is that you can't use word-delimiting characters inside annotations. The whole string would be treated as two tokens which would result in unexpected behaviour.

The InlineAnnotationFilter can be slightly customized.

List of supported options

• start - start delimiter for inline annotation
• end - end delimiter for inline annotation
• prefix - string to be prepended to synonym, that is created from inline annotation
• suffix - string to be apended to synonym, that is created from inline annotation
• token-type - token type of synonym
• delimiter - delimiter for multiple inline annotations

Example providing default values

index :
    analysis :
        analyzer :                
            annotation :
                type : annotation
                start : [
                end : ]
                prefix : [
                suffix : ]
                token-type: synonym
                delimiter : ;

This plugin was successfuly tested on elasticsearch version 0.90.2