Implement "concept" as a first-class citizen along-side Term. about resin HOT 13 CLOSED

Also on that subject that's a reason to seperate the analyzer and tokenizer into seperate interfaces, especially in regards to concept identification where the concept can span/encapsulate multiple terms. Using an analyzer with a tightly coupled tokenization function makes this a nightmare

from resin.

check out https://github.com/jhashemi/resin/tree/master/src/Resin.Analyses.Concept

ideally concepts are represented as graphs. Most definitely a separate concept index will be needed. A implementation of IVocabulary that depends on a resin index will be needed. This is very very rudimentary and untested, I just wanted to get my ideas to paper

from resin.

Can this also be used for synonyms?

from resin.

This is instead of synonyms. "King" and "emperor" should both be part of the same (oppressive and undemocratic ruler) concept.

from resin.

Concepts also span multiple terms, then you have to deal with disambiguation ;-)

I'll give a very very basic go and submit a PR

from resin.

@jhashemi you're giving this a go?

I could tell you about my ideas but I'm not going to. You seem to have an itch. Just a basic proof of concept will do :)

from resin.

Before i go deeper into it wanted to chat about knowledge base or ontology base or api based.

Maybe ill uml it out and push an architecture project.

from resin.

Sounds good.

Some thought on this issue. When I added it I was thinking about (1) how to implement word2vec, simplified in the same way the vector space model is simplified in Resin and in Lucene. But also (2) how to produce word vectors at indexing time. What if you add one document at a time to your index, how would you then be able to produce word vectors? It seems not possible. So perhaps "concepts" or word vectors or sentiment analysis or whatever you want to call it is an operation you do on an existing index. The sentiment analysis operation could produce a new concept-based index that complements the term-based one.

The concept-based index would contain pointers into the term-based index which in turn has pointers into the postings and document store.

Having a concept-based index would mean you could make more directed lookups into the term-based index instead of large scans.

All in theory and a bit diffuse in my mind at the moment.

Also, we would need a new tree to represent words instead of just characters. Does it have to be a B+ tree? I mean sure, all devs should roll a B+ tree once in their lives, I guess. Maybe it'll be fun?

from resin.

This looks pretty good: https://github.com/asengupta/BPlusTree/blob/master/BPlusTree/BTreeNode.cs

from resin.

Also for graphs, typically a sparse adjacency matrix implementation works best. with each axis being your node ID's and relationships established as 0 or 1. You can use a Bitmap Index to make traversal extremely fast.

from resin.

I will check this out shortly. I ran through the code and it looked very promising.

from resin.

This issue is still open but needs a new strategy because of the new type of index introduced here: 5f85425

from resin.

Will be solved at a later time.

from resin.