sbenthall / bluestocking Goto Github PK

in the preprocessing step, replace pronouns with the chunks

Build demo functionality:

• given a text
• tokenize the text, look up every word in wikipedia
• build knowledge base based on collected articles
• evaluate consistency of original text with knowledge base
• segment into: supported claims, new claims, and contradicted claims.

Introduce Concept class as wrapper around a semantic node. A Concept has:

• a set of terms that elicit it (e.g. a single term, or set of words in a WordNet synset)
• a reference to the knowledge base they are a part of (see Ned Block on Conceptual Role Semantics)
• a method that returns the available relations for this concept, in the knowledge base and/or based on wordnet (antonyms)

add chunking in the preprocessing step

script the functionality of: given a word (or article title), grab the wikipedia article on the subject, and strip markup to return text.

For now, don't worry about marking the articles for different handling on the Concept level.

Currently, knowledge bases are just built by appending the relations parsed from individual documents in the corpus.

This can lead to the introduction of contradictory relations into the knowledge base.

One way to deal with this would be to check for consistency between Concepts when adding relations to a knowledge base. If the concept in a newly added set of relations is similar enough to an existing concept, then the two concepts can be merged and all relations applicable to either can be adopted. If the concepts are to deviant (despite, say, having the same triggering word), then the new concept can be preserved separately.

The parser is pulling out tautological relations like:

[(True, 'yesterday', 'yesterday'), (True, 'today', 'today')]

which are throwing off consistency scoring.

this can go in the bin/ directory at the top level of the package. see:

http://guide.python-distribute.org/creation.html#arranging-your-file-and-directory-structure

Bring in Dave's shallow inference system based on wordnet.

The parser seems to be stripping the first character from some tokens. The knowledge base for "Today was not good. Ice cream is not good. Spinach is bad. I hate everything."

...includes relations like " (False, 'ice', 'ood')" (not missing 'g')

After chunking, parse within chunks for relations. Only establish relations between chunks, and internal to chunks.

delimit combination/cooccurance detection to within clauses

include requirements in a proper requirements file
package for easy installation with pip