louismullie / treat Goto Github PK

Natural language processing framework for Ruby.

License: Other

Ruby 100.00%

treat's Introduction

New in v2.0.5: OpenNLP integration and Yomu support

Treat is a toolkit for natural language processing and computational linguistics in Ruby. The Treat project aims to build a language- and algorithm- agnostic NLP framework for Ruby with support for tasks such as document retrieval, text chunking, segmentation and tokenization, natural language parsing, part-of-speech tagging, keyword extraction and named entity recognition. Learn more by taking a quick tour or by reading the manual.

Features

Text extractors for PDF, HTML, XML, Word, AbiWord, OpenOffice and image formats (Ocropus).
Text chunkers, sentence segmenters, tokenizers, and parsers (Stanford & Enju).
Lexical resources (WordNet interface, several POS taggers for English).
Language, date/time, topic words (LDA) and keyword (TF*IDF) extraction.
Word inflectors, including stemmers, conjugators, declensors, and number inflection.
Serialization of annotated entities to YAML, XML or to MongoDB.
Visualization in ASCII tree, directed graph (DOT) and tag-bracketed (standoff) formats.
Linguistic resources, including language detection and tag alignments for several treebanks.
Machine learning (decision tree, multilayer perceptron, LIBLINEAR, LIBSVM).
Text retrieval with indexation and full-text search (Ferret).

Contributing

I am actively seeking developers that can help maintain and expand this project. You can find a list of ideas for contributing to the project here.

Authors

Lead developper: @louismullie [Twitter]

Contributors:

@bdigital
@automatedtendencies
@LeFnord
@darkphantum
@whistlerbrk
@smileart
@erol

License

This software is released under the GPL License and includes software released under the GPL, Ruby, Apache 2.0 and MIT licenses.

treat's People

Contributors

Stargazers

Watchers

Forkers

bobbytables kshahkshah wsh robadler lmiranda darkphantum zerox-tj ruflin colinsurprenant jammie turadg hkf ianstalter gouthamvel akmassey smileart smileart-levelup psychoanalytix inem jayzheng erol wstrinz geoffharcourt dannyoon jhund ajaila timwhite47 kamkha maxdavila zackliscio svankie washu bckwon83 azizur77 ngpestelos elitmuscasts pac darkslategrey watashi-wa prashanthrajagopal pchevry chrisanderton lefnord uttamkgulshan opencoderx morred dalton rakeshpatri adammcarth spiliero kkooff2000 alive-attemptthelifeganghouse kathgironpe diasks2 sagarjunnarkar hackur mjason lanstro rutikasb domgetter chagge dhootha baiming969696 conorhastings slacken femto osxi jevenwill yeyong123 jbranchaud ojak c4eo zpvip faustoct1 trosborn dancomand x44x45x41x4e siyuan1990 shyammohankanojia zzmjohn danoph geluomi nofxx thcrock shanemwillson guidedlabs arpit anukat2015 whilei zotherstupidguy irfus davidamitchell hcyjason indentlabs misakiyui samuelcodes j-mcnally nixswinner pierrediderot slimbn

treat's Issues

Implosion/to_s problem with Enclitics

    text = "It's about time."
    text = sentence(text).apply(:tokenize, :parse)
    puts text.to_s

Results in:

It 's about time.

Should that to_s without the extra space between It and `s?

Stanford parser: wrong results for german, no error message

The stanford parser gives wrong results for german but no error message:

  # ruby encoding: utf-8
  ENV['JAVA_HOME'] = "/opt/java"
  ENV['LD_LIBRARY_PATH'] = "/opt/java/jre/lib/amd64"
  require 'treat'
  include Treat::Core::DSL
  Treat.core.language.default = 'german' 
  Treat.core.verbosity.debug

  s = sentence 'Der wilde Kerl lebte in einem gelben Haus.'
  s.do(:tokenize, :parse)
  s.print_tree

Output:

+ Sentence (15019540)  --- "(ART Der) lebte (APPR in)."  ---  {:tag_set=>:stutgart}   --- [] 
|
+--> Symbol (14993200)  --- "(ART Der)"  ---  {:tag=>"NP-SB"}   --- [] 
+--> Word (15125440)  --- "lebte"  ---  {:tag=>"VVFIN"}   --- [] 
+--> Symbol (13203700)  --- "(APPR in)"  ---  {:tag=>"PP"}   --- [] 
+--> Punctuation (14970040)  --- "."  ---  {:tag=>"$."}   --- []

==> The first word is no Symbol, and there are several words missing.

English is working fine.
Using the stanford-core-nlp gem everything looks good:

  # ruby encoding: utf-8
  ENV['JAVA_HOME'] = "/opt/java"
  ENV['LD_LIBRARY_PATH'] = "/opt/java/jre/lib/amd64"
  require 'stanford-core-nlp'
  StanfordCoreNLP.use :german
  pipeline =  StanfordCoreNLP.load(:tokenize, :ssplit, :parse)
  text = StanfordCoreNLP::Annotation.new('Der wilde Kerl lebte in einem gelben Haus.')
  pipeline.annotate(text)

  text.get(:sentences).each do |sentence|
    puts sentence.get(:basic_dependencies).to_s
    sentence.get(:tokens).each do |token|
      puts "#{token.get(:original_text)} #{token.get(:part_of_speech)}"
    end
  end

Output:

Der ART
wilde ADJA
Kerl NN
lebte VVFIN
in APPR
einem ART
gelben ADJA
Haus NN
. $.

If I try to use the stanford-core-nlp gem after Treats do-method, I get an error with bind-it:

# ruby encoding: utf-8
ENV['JAVA_HOME'] = "/opt/java"
ENV['LD_LIBRARY_PATH'] = "/opt/java/jre/lib/amd64"
require 'treat'
include Treat::Core::DSL
Treat.core.language.default = 'german' 
Treat.core.verbosity.debug

s = sentence 'Der wilde Kerl lebte in einem gelben Haus.'
s.do(:tokenize, :parse)

require 'stanford-core-nlp'
StanfordCoreNLP.use :german
pipeline =  StanfordCoreNLP.load(:tokenize, :ssplit, :parse)
text = StanfordCoreNLP::Annotation.new('Der wilde Kerl lebte in einem gelben Haus.')
pipeline.annotate(text)

Error:

/home/klippstein/.rvm/gems/ruby-1.9.3-p392/gems/bind-it-0.2.7/lib/bind-it/rjb_proxy.rb:37:in `method_missing': unknown exception (NullPointerException)
  from /home/klippstein/.rvm/gems/ruby-1.9.3-p392/gems/bind-it-0.2.7/lib/bind-it/rjb_proxy.rb:37:in `method_missing'
  from /home/klippstein/Dropbox/Ruby-AptanaWorkspace/spielwiese/treat.rb:28:in `<main>'

Encodings

Identify which algorithms are encoding-dependent and which are not.
When possible, enlarge the number of encodings functioning with each algorithm.
Integrate an encoding guesser for external files.
Make sure the encoding of a document is reflected when serialized to XML.

edu.stanford.nlp.io.RuntimeIOException during :parse

I am getting an exception when I do

  para = Paragraph meta.text_summary
  p_text = para.do(:segment, :parse)  #error on :parse, :segment works fine

I have tried setting the path to english-left3words-distsim.tagger in the classpath env variable.
That did not help

Error:
edu.stanford.nlp.io.RuntimeIOException: ERROR: cannot find properties file "{pos.model=/home/varun/.rvm/gems/ruby-1.9.3-p286@global/gems/treat-1.2.0/lib/treat/../../models/stanford/taggers/english-left3words-distsim.tagger, parser.model=/home/varun/.rvm/gems/ruby-1.9.3-p286@global/gems/treat-1.2.0/lib/treat/../../models/stanford/grammar/englishPCFG.ser.gz, annotators=tokenize, ssplit, pos, lemma, parse}" in the classpath!

link parser integration

The gem is available at github: https://github.com/ged/linkparser

It is much easier (at least for me) to install the required C libraries (they are part of AbiWord) than Enju parser.

Wrong conversion verb to infinitive

Hello guys. It seems to me that some of words don't change to infinitive correct.

'walked'.infinitive - walke
'neglected'.infinitive - neglecte
'earned'.infinitive - earne

But 'running'.infinitive - run. Some other words also.
What am I doing wrong?

NoMethodError for the segment method

When I try and call segment on a paragraph, I get this error:

NoMethodError: undefined method `segment' for Treat::Workers::Processors::Segmenters::Scalpel:Class

Here is the call that gives the error (this is right from the documentation btw):
p = paragraph('A walk in the park. A trip on a boat.').segment

Checked the issues and didn't see anything like this...hopefully it hasn't been asked before.

Thanks

Why the strange syntax?

For example, Treat:

uses Word "confusing" instead of Word.new "confusing" or word "confusing"
uses a method called do -- different from the Ruby keyword of the same name.

Why? Is there a rationale? I find it to be a mismatch with typical Ruby style.

Add support for SQL databases

Custom extractors

I am trying to build quick prototype of query understanding like on http://adioso.com/ using treat.

I decided to write own extractors to detect some groups of patterns.
In the documentations you show how to extend Treat by adding custom stemmers & tokenizers. Is there a way to add custom extractor similarly?
I had to edit gem sources to play with it, but I wish I could extend it via initializer in my rails app.

Here's what I have: https://gist.github.com/inem/5396787
This one is very primitive. I am thinking to write some extractors, who will be able to search against app database. To do so, these extractors should not be the part of treat gem definitely.

Or maybe I am doing it completely wrong way?

Create a builder class that includes Treat::Core::DSL

IllegalArgumentException

Hi Louis,

I want to parse a german document,
that's why I do:

Treat.core.language.detect = true

and

doc = Treat::Entities::Paragraph.build german_text
text.do :segment, :parse, :category

and here comes the message:

IllegalArgumentException Unknown option: -retainTmpSubcategories

Another thing,
can you publish your future plans for treat,
may some more people could help?

LoadError: cannot load such file -- ruby-readability

upon installing the gem, the following line of code, produced the above error.

d = document 'http://en.wikipedia.org/wiki/Sack_of_Amorium'

After a quick google, installing this gem solves the issue. Would the solution just be to add ruby-readability as a runtime dependency in the gemspec? If so, I'd be glad to submit a PR.

debug method overiding debug method in rails

Hi Louis
Another little bug, when I want to use treat with rails, I have an error method when I load stylesheet_link_tag

<%= stylesheet_link_tag "application", :media => "all" %>
<%= javascript_include_tag "application" %>

I have this trace :
The debug method of treat is overiding the debug method in rails I think .

wrong number of arguments (0 for 1)
Extracted source (around line #11):

8:
9:
10:
11: <%= stylesheet_link_tag "application", :media => "all" %>
12: <%= javascript_include_tag "application" %>
13:
14:

Application Trace | Framework Trace | Full Trace
treat (1.0.4) lib/treat/kernel.rb:184:in debug' actionpack (3.2.3) lib/sprockets/helpers/rails_helper.rb:83:indebug_assets?'
actionpack (3.2.3) lib/sprockets/helpers/rails_helper.rb:39:in stylesheet_link_tag' actionpack (3.2.3) lib/action_view/template.rb:143:inblock in render'
activesupport (3.2.3) lib/active_support/notifications.rb:125:in instrument' actionpack (3.2.3) lib/action_view/template.rb:141:inrender'
actionpack (3.2.3) lib/action_view/renderer/template_renderer.rb:59:in render_with_layout' actionpack (3.2.3) lib/action_view/renderer/template_renderer.rb:45:inrender_template'
actionpack (3.2.3) lib/action_view/renderer/template_renderer.rb:18:in render' actionpack (3.2.3) lib/action_view/renderer/renderer.rb:36:inrender_template'
haml (3.1.5) lib/haml/helpers/rails_323_textarea_fix.rb:26:in render_template_with_haml' actionpack (3.2.3) lib/action_view/renderer/renderer.rb:17:inrender'
actionpack (3.2.3) lib/abstract_controller/rendering.rb:110:in _render_template' actionpack (3.2.3) lib/action_controller/metal/streaming.rb:225:in_render_template'
actionpack (3.2.3) lib/abstract_controller/rendering.rb:103:in render_to_body' haml (3.1.5) lib/haml/helpers/rails_323_textarea_fix.rb:14:inrender_to_body_with_haml'
actionpack (3.2.3) lib/action_controller/metal/renderers.rb:28:in render_to_body' actionpack (3.2.3) lib/action_controller/metal/compatibility.rb:50:inrender_to_body'
actionpack (3.2.3) lib/abstract_controller/rendering.rb:88:in render' actionpack (3.2.3) lib/action_controller/metal/rendering.rb:16:inrender'
actionpack (3.2.3) lib/action_controller/metal/instrumentation.rb:40:in block (2 levels) in render' activesupport (3.2.3) lib/active_support/core_ext/benchmark.rb:5:inblock in ms'
/Users/ztera/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/benchmark.rb:310:in realtime' activesupport (3.2.3) lib/active_support/core_ext/benchmark.rb:5:inms'
actionpack (3.2.3) lib/action_controller/metal/instrumentation.rb:40:in block in render' actionpack (3.2.3) lib/action_controller/metal/instrumentation.rb:83:incleanup_view_runtime'
actionpack (3.2.3) lib/action_controller/metal/instrumentation.rb:39:in render' remotipart (1.0.2) lib/remotipart/render_overrides.rb:8:inrender'
actionpack (3.2.3) lib/action_controller/metal/implicit_render.rb:10:in default_render' actionpack (3.2.3) lib/action_controller/metal/implicit_render.rb:5:insend_action'
actionpack (3.2.3) lib/abstract_controller/base.rb:167:in process_action' actionpack (3.2.3) lib/action_controller/metal/rendering.rb:10:inprocess_action'
actionpack (3.2.3) lib/abstract_controller/callbacks.rb:18:in block in process_action' activesupport (3.2.3) lib/active_support/callbacks.rb:425:in_run__563304916674385327__process_action__276644419250049832__callbacks'
activesupport (3.2.3) lib/active_support/callbacks.rb:405:in __run_callback' activesupport (3.2.3) lib/active_support/callbacks.rb:385:in_run_process_action_callbacks'
activesupport (3.2.3) lib/active_support/callbacks.rb:81:in run_callbacks' actionpack (3.2.3) lib/abstract_controller/callbacks.rb:17:inprocess_action'
actionpack (3.2.3) lib/action_controller/metal/rescue.rb:29:in process_action' actionpack (3.2.3) lib/action_controller/metal/instrumentation.rb:30:inblock in process_action'
activesupport (3.2.3) lib/active_support/notifications.rb:123:in block in instrument' activesupport (3.2.3) lib/active_support/notifications/instrumenter.rb:20:ininstrument'
activesupport (3.2.3) lib/active_support/notifications.rb:123:in instrument' actionpack (3.2.3) lib/action_controller/metal/instrumentation.rb:29:inprocess_action'
actionpack (3.2.3) lib/action_controller/metal/params_wrapper.rb:205:in process_action' actionpack (3.2.3) lib/abstract_controller/base.rb:121:inprocess'
actionpack (3.2.3) lib/abstract_controller/rendering.rb:45:in process' actionpack (3.2.3) lib/action_controller/metal.rb:203:indispatch'
actionpack (3.2.3) lib/action_controller/metal/rack_delegation.rb:14:in dispatch' actionpack (3.2.3) lib/action_controller/metal.rb:246:inblock in action'
actionpack (3.2.3) lib/action_dispatch/routing/route_set.rb:73:in call' actionpack (3.2.3) lib/action_dispatch/routing/route_set.rb:73:indispatch'
actionpack (3.2.3) lib/action_dispatch/routing/route_set.rb:36:in call' journey (1.0.3) lib/journey/router.rb:68:inblock in call'
journey (1.0.3) lib/journey/router.rb:56:in each' journey (1.0.3) lib/journey/router.rb:56:incall'
actionpack (3.2.3) lib/action_dispatch/routing/route_set.rb:600:in call' rack-pjax (0.5.9) lib/rack/pjax.rb:12:incall'
mongoid (2.4.10) lib/rack/mongoid/middleware/identity_map.rb:33:in block in call' mongoid (2.4.10) lib/mongoid.rb:133:inunit_of_work'
mongoid (2.4.10) lib/rack/mongoid/middleware/identity_map.rb:33:in call' sass (3.1.18) lib/sass/plugin/rack.rb:54:incall'
warden (1.1.1) lib/warden/manager.rb:35:in block in call' warden (1.1.1) lib/warden/manager.rb:34:incatch'
warden (1.1.1) lib/warden/manager.rb:34:in call' actionpack (3.2.3) lib/action_dispatch/middleware/best_standards_support.rb:17:incall

Use Enumators

The API is confusing enough regarding traversing an entity tree (docs might help) but it's really a let down that the iterations aren't actually using the Ruby Enumerable module and Enumerator class.

1.3 - OpenNLP integration

This involves finishing the open-nlp gem at https://github.com/louismullie/open-nlp, and writing some interfaces to the tokenizers, segmenters and chunkers, as well as any other useful too fitting under the library.

Word humongous

Every time I try to input a string text containing the word 'humongous' I get the following exception below:

/gems/treat-2.0.4/lib/treat/entities/entity/buildable.rb:24 8:in `from_serialized_file': Path 'humongous (Treat::Exception)
' does not point to a readable file.

Tested with:

"In 1991, the 66 year old money manager hit a bonanza,racking up a humongous 88.8% total return more than twice the return of the average growth fund and nearly three times the SP 500-stock index."

"humongous amounts of money"

What could be the error and how would I be able to resolve this?
Thanks.

undefined method `children' for nil:NilClass (NoMethodError)

I'm trying to parse a document, but I get this error. Here's my code attempt

require 'treat'
include Treat::Core::DSL

rovio_privacy = document 'http://www.rovio.com/privacy'
rovio_privacy.apply :chunk, :segment, :tokenize, :parse

And here's the stacktrace

/Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/core/dsl.rb:17:in `method_missing': undefined method `children' for nil:NilClass (NoMethodError)
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/workers/processors/parsers/stanford.rb:57:in `recurse'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/workers/processors/parsers/stanford.rb:50:in `parse'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/delegatable.rb:66:in `call_worker'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/delegatable.rb:40:in `block (2 levels) in add_workers'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:41:in `do_task'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:49:in `block in do_task'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:48:in `each'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:48:in `do_task'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:49:in `block in do_task'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:48:in `each'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:48:in `do_task'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:49:in `block in do_task'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:48:in `each'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:48:in `do_task'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:22:in `block in apply'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:6:in `each'
    from /Users/gabro/.rvm/gems/ruby-1.9.3-p448/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:6:in `apply'
    from test.rb:10:in `<main>'

I used the automatic installer and manually installed Enju, which is in my PATH.

Any idea on why I'm getting such error?

TF_IDF issue with gem version, but Collection issue with master branch

I'm trying to work TF_IDF out. If I use the gem install version, I cannot run c.words[0].tf_idf as per another ticket.

If I use your master branch, I cannot initialize a Collection. In rails console I do (which used to work in the gem install version):

include Treat
include Treat::Entities

c = Collection('data') # which fails...

Improve documentation

While it's nice to see "you can do this" (e.g. sentance(word("..."))), the docs don't make it clear what I should expect to happen. Right after introducing building, it'd be a good place to talk about visualizations and include demos.

The traversal docs could really stand to be expanded - the example from issue #35 would be useful, if expanded with an example tree and #=> comments about what's yielded.

punkt not found

While running

Treat::Core::Installer.install 'english' in Ruby 1.9.3p194, I get

RuntimeError: Couldn't download http://www.louismullie.com/treat/punkt/english.yaml (Max number of attempts reached). Error: (Response code was not 200 , but 404.)

http://www.louismullie.com/treat/ shows no punkt directory. Should we just follow the manual install as laid out in Manual Installation?

Install as normal user

Hi there,

I just tried installing treat on Arch linux and basically was completely unable to get it to work.
As my normal user, I did gem install treat, which worked as expected. If I then do Treat.install inside IRB, I get the following error:

rirb(main):001:0> require 'treat'
=> true
irb(main):002:0> Treat.install

Treat Installer, v. 1.0.6


1. Install language-independent gem dependencies.

Do you want to :
1 - Install all default language-independent dependencies
2 - Select dependencies to install manually
3 - Skip this step 

Please enter one of 1, 2, 3: 
1


- Installing psych to (un)serialize annotated entities to YAML format.
Gem::FilePermissionError: You don't have write permissions into the /usr/lib/ruby/gems/1.9.1 directory.
    from /usr/lib/ruby/1.9.1/rubygems/installer.rb:448:in `verify_gem_home'
    from /usr/lib/ruby/1.9.1/rubygems/installer.rb:147:in `install'
    from /usr/lib/ruby/1.9.1/rubygems/dependency_installer.rb:297:in `block in install'
    from /usr/lib/ruby/1.9.1/rubygems/dependency_installer.rb:270:in `each'
    from /usr/lib/ruby/1.9.1/rubygems/dependency_installer.rb:270:in `each_with_index'
    from /usr/lib/ruby/1.9.1/rubygems/dependency_installer.rb:270:in `install'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/installer.rb:302:in `block in install_gem'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/kernel.rb:25:in `call'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/kernel.rb:25:in `silence_warnings'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/installer.rb:301:in `install_gem'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/installer.rb:141:in `block in install_dependencies'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/installer.rb:139:in `each'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/installer.rb:139:in `install_dependencies'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/installer.rb:62:in `install'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat.rb:56:in `install'
    from (irb):2
    from /usr/bin/irb:12:in `<main>'irb(main):003:0>

The installer shouldn't try to install those files in /usr, after all the gem resides in ~/.gem.

If I do the same thing as root, the gem gets installed in /root/.gem, the install procedure works. But afterwords I am unable to use treat as a normal user, because it was installed into `/root/.gem'. Installing the gem as my user as well didn't work either, following error:

irb(main):001:0> require 'treat'
=> true
irb(main):002:0> "test sentence is failing".parse
RuntimeError: Could not find JAR file (looking in /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/bin/stanford/joda-time.jar).
    from /usr/lib/ruby/gems/1.9.1/gems/bind-it-0.2.0/lib/bind-it/jar_loader.rb:39:in `load'
    from /usr/lib/ruby/gems/1.9.1/gems/bind-it-0.2.0/lib/bind-it/binding.rb:56:in `load_jar'
    from /usr/lib/ruby/gems/1.9.1/gems/bind-it-0.2.0/lib/bind-it/binding.rb:64:in `block in load_default_jars'
    from /usr/lib/ruby/gems/1.9.1/gems/bind-it-0.2.0/lib/bind-it/binding.rb:61:in `each'
    from /usr/lib/ruby/gems/1.9.1/gems/bind-it-0.2.0/lib/bind-it/binding.rb:61:in `load_default_jars'
    from /usr/lib/ruby/gems/1.9.1/gems/bind-it-0.2.0/lib/bind-it/binding.rb:48:in `bind'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/loaders/stanford.rb:35:in `load'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/processors/parsers/stanford.rb:62:in `init'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/processors/parsers/stanford.rb:28:in `parse'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/entities/abilities/delegatable.rb:69:in `call_worker'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/entities/abilities/delegatable.rb:39:in `block (2 levels) in add_workers'
    from /home/profmaad/.gem/ruby/1.9.1/gems/treat-1.0.6/lib/treat/proxies.rb:14:in `method_missing'
    from (irb):2
    from /usr/bin/irb:12:in `<main>'

So the installer installs system-wide, but the gem then looks locally for the files.

Please fix this, I am completely unable to use treat because of this. It seems to me that the installer should be installing where the gem is located.

Natural Language Generation

I read through the manual and I'm curious, is there any way to use treat to generate English sentences from a set of data? For example if I had a set of weather or sports statistics and I wanted to auto generate an English summary.

If this is something not currently included, is there a good java or python library that can be blatantly ripped off? I think it would be a killer feature. looked through the NLTK Documentation but I didn't find a clear way of doing this. Maybe I just missed it though.

Treat being loaded causes slowness in stanford-core-nlp

When treat is loaded in my Gemfile, stanford-core-nlp runs substantially slower.

Here's the code that runs quite slow. I don't think it's anything very complex.

text.get(:sentences).each do |sentence|
  sentence.get(:tokens).each do |token|
    value =  token.get(:original_text).to_s
    pos = token.get(:part_of_speech).to_s
    et = token.get(:named_entity_tag).to_s
    ner_list << [et,value]
    insert_unique(pos_hash, pos, value)
  end
end

I've enclosed a couple of profile trace images that shows the same code before and after loading treat.

Hope this helps and apologies if it's down to my ignorance.

Regards,

Neil

With treat:

Without treat:

Treat::Core::Installer.install returns LoadError

First, thanks for a great piece of software.
Lately I upgraded to treat 2.0.5.

Since then, I get the following error when running, 'require treat' and 'Treat::Core::Installer.install' in IRB
LoadError: cannot load such file -- stanford-core-nlp
from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:45:in require' from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:45:inrequire'
from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat/loaders/stanford.rb:6:in <class:Stanford>' from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat/loaders/stanford.rb:4:in<top (required)>'
from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:45:in require' from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:45:inrequire'
from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat/autoload.rb:22:in block in included' from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat/autoload.rb:22:ineach'
from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat/autoload.rb:22:in included' from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat/modules.rb:13:ininclude'
from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat/modules.rb:13:in <module:Loaders>' from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat/modules.rb:13:inmodule:Treat'
from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat/modules.rb:1:in <top (required)>' from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat.rb:20:inrequire_relative'
from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat.rb:20:in <module:Treat>' from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/treat-2.0.5/lib/treat.rb:14:in<top (required)>'
from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:110:in require' from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:110:inrescue in require'
from /home/jaj/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:35:in require' from (irb):1 from /home/jaj/.rbenv/versions/1.9.3-p194/bin/irb:12:in

I haven't been able to find the cause of the problem

Best, Jakob

Trying to tag...

LoadError: cannot load such file -- engtagger
from /home/dave/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require' from /home/dave/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/lexicalizers/taggers/lingua.rb:18:in block in <class:Lingua>' from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/kernel.rb:25:incall'
from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/kernel.rb:25:in silence_warnings' from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/lexicalizers/taggers/lingua.rb:18:inclass:Lingua'
from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/lexicalizers/taggers/lingua.rb:15:in <top (required)>' from /home/dave/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
from /home/dave/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require' from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/groupable.rb:14:inconst_missing'
from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/groupable.rb:73:in const_get' from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/groupable.rb:73:inconst_get'
from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/entities/abilities/delegatable.rb:65:in call_worker' from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/entities/abilities/delegatable.rb:39:inblock (2 levels) in add_workers'
from /home/dave/.rvm/gems/ruby-1.9.3-p194/gems/treat-1.0.6/lib/treat/proxies.rb:14:in `method_missing'
from (irb):21

Is there another step towards installation tha twould include the tagger?

segfault when :parsing w/ rjb 1.4.5

After updating to rjb 1.4.5 this happened when I tried to :parse a document. I have to admit, this goes over my head, but I think it's an issue introduced with rjb 1.4.5, since it works fine with rjb 1.4.3. If you need any other info, let me know! Hope this is useful.

My steps (in IRB):

require 'treat'
include Treat::Core::DSL
d = document("http://www.guardian.co.uk/uk/2012/oct/22/cardiff-man-court-charged-murder")
d.apply :chunk, :segment, :parse

The result:

1.9.3p362 :005 > d.apply :chunk, :segment, :parse
Invalid memory access of location 0x0 rip=0x1088484ed
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/jar_loader.rb:45: [BUG] Segmentation fault
ruby 1.9.3p362 (2012-12-25 revision 38607) [x86_64-darwin12.2.0]

-- Control frame information -----------------------------------------------
c:0057 p:---- s:0239 b:0239 l:000238 d:000238 CFUNC  :load
c:0056 p:0028 s:0234 b:0234 l:000233 d:000233 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/jar_loader.rb:45
c:0055 p:0029 s:0231 b:0231 l:000230 d:000230 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/jar_loader.rb:38
c:0054 p:0117 s:0226 b:0226 l:000225 d:000225 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/jar_loader.rb:27
c:0053 p:0026 s:0221 b:0221 l:000220 d:000220 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/binding.rb:63
c:0052 p:0056 s:0216 b:0216 l:000205 d:000215 BLOCK  /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/binding.rb:71
c:0051 p:---- s:0211 b:0211 l:000210 d:000210 FINISH
c:0050 p:---- s:0209 b:0209 l:000208 d:000208 CFUNC  :each
c:0049 p:0017 s:0206 b:0206 l:000205 d:000205 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/binding.rb:68
c:0048 p:0072 s:0203 b:0203 l:000202 d:000202 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/binding.rb:55
c:0047 p:0061 s:0200 b:0200 l:000199 d:000199 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/stanford-core-nlp-0.5.1/lib/stanford-core-nlp.rb:125
c:0046 p:0020 s:0197 b:0197 l:000196 d:000196 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/stanford-core-nlp-0.5.1/lib/stanford-core-nlp.rb:140
c:0045 p:0163 s:0190 b:0187 l:000186 d:000186 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/processors/parsers/stanford.rb:74
c:0044 p:0049 s:0181 b:0181 l:000180 d:000180 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/processors/parsers/stanford.rb:33
c:0043 p:0190 s:0172 b:0172 l:000171 d:000171 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/delegatable.rb:66
c:0042 p:0101 s:0163 b:0163 l:000720 d:000162 LAMBDA /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/delegatable.rb:40
c:0041 p:---- s:0159 b:0159 l:000158 d:000158 FINISH
c:0040 p:0091 s:0157 b:0157 l:000156 d:000156 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:41
c:0039 p:0023 s:0148 b:0148 l:000139 d:000147 BLOCK  /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:49
c:0038 p:---- s:0145 b:0145 l:000144 d:000144 FINISH
c:0037 p:---- s:0143 b:0143 l:000142 d:000142 CFUNC  :each
c:0036 p:0129 s:0140 b:0140 l:000139 d:000139 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:48
c:0035 p:0023 s:0131 b:0131 l:000122 d:000130 BLOCK  /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:49
c:0034 p:---- s:0128 b:0128 l:000127 d:000127 FINISH
c:0033 p:---- s:0126 b:0126 l:000125 d:000125 CFUNC  :each
c:0032 p:0129 s:0123 b:0123 l:000122 d:000122 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:48
c:0031 p:0023 s:0114 b:0114 l:000105 d:000113 BLOCK  /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:49
c:0030 p:---- s:0111 b:0111 l:000110 d:000110 FINISH
c:0029 p:---- s:0109 b:0109 l:000108 d:000108 CFUNC  :each
c:0028 p:0129 s:0106 b:0106 l:000105 d:000105 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:48
c:0027 p:0173 s:0097 b:0097 l:000085 d:000096 BLOCK  /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:22
c:0026 p:---- s:0091 b:0091 l:000090 d:000090 FINISH
c:0025 p:---- s:0089 b:0089 l:000088 d:000088 CFUNC  :each
c:0024 p:0012 s:0086 b:0086 l:000085 d:000085 METHOD /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:6
c:0023 p:0017 s:0082 b:0082 l:000338 d:000081 EVAL   (irb):5
c:0022 p:---- s:0080 b:0080 l:000079 d:000079 FINISH
c:0021 p:---- s:0078 b:0078 l:000077 d:000077 CFUNC  :eval
c:0020 p:0028 s:0071 b:0071 l:000070 d:000070 METHOD /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/workspace.rb:80
c:0019 p:0033 s:0064 b:0063 l:000062 d:000062 METHOD /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/context.rb:254
c:0018 p:0031 s:0058 b:0058 l:001238 d:000057 BLOCK  /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:159
c:0017 p:0042 s:0050 b:0050 l:000049 d:000049 METHOD /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:273
c:0016 p:0011 s:0045 b:0045 l:001238 d:000044 BLOCK  /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:156
c:0015 p:0144 s:0041 b:0041 l:000024 d:000040 BLOCK  /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/ruby-lex.rb:243
c:0014 p:---- s:0038 b:0038 l:000037 d:000037 FINISH
c:0013 p:---- s:0036 b:0036 l:000035 d:000035 CFUNC  :loop
c:0012 p:0009 s:0033 b:0033 l:000024 d:000032 BLOCK  /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/ruby-lex.rb:229
c:0011 p:---- s:0031 b:0031 l:000030 d:000030 FINISH
c:0010 p:---- s:0029 b:0029 l:000028 d:000028 CFUNC  :catch
c:0009 p:0023 s:0025 b:0025 l:000024 d:000024 METHOD /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/ruby-lex.rb:228
c:0008 p:0046 s:0022 b:0022 l:001238 d:001238 METHOD /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:155
c:0007 p:0011 s:0019 b:0019 l:001668 d:000018 BLOCK  /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:70
c:0006 p:---- s:0017 b:0017 l:000016 d:000016 FINISH
c:0005 p:---- s:0015 b:0015 l:000014 d:000014 CFUNC  :catch
c:0004 p:0183 s:0011 b:0011 l:001668 d:001668 METHOD /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:69
c:0003 p:0142 s:0006 b:0006 l:0020b8 d:0020b8 EVAL   /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/bin/irb:16
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:0020b8 d:0020b8 TOP   

-- Ruby level backtrace information ----------------------------------------
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/bin/irb:16:in `<main>'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:69:in `start'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:69:in `catch'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:70:in `block in start'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:155:in `eval_input'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/ruby-lex.rb:228:in `each_top_level_statement'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/ruby-lex.rb:228:in `catch'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/ruby-lex.rb:229:in `block in each_top_level_statement'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/ruby-lex.rb:229:in `loop'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/ruby-lex.rb:243:in `block (2 levels) in each_top_level_statement'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:156:in `block in eval_input'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:273:in `signal_status'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb:159:in `block (2 levels) in eval_input'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/context.rb:254:in `evaluate'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/workspace.rb:80:in `evaluate'
/Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/workspace.rb:80:in `eval'
(irb):5:in `irb_binding'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:6:in `apply'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:6:in `each'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:22:in `block in apply'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:48:in `do_task'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:48:in `each'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:49:in `block in do_task'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:48:in `do_task'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:48:in `each'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:49:in `block in do_task'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:48:in `do_task'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:48:in `each'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:49:in `block in do_task'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb:41:in `do_task'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/delegatable.rb:40:in `block (2 levels) in add_workers'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/delegatable.rb:66:in `call_worker'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/processors/parsers/stanford.rb:33:in `parse'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/processors/parsers/stanford.rb:74:in `init'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/stanford-core-nlp-0.5.1/lib/stanford-core-nlp.rb:140:in `load'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/stanford-core-nlp-0.5.1/lib/stanford-core-nlp.rb:125:in `bind'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/binding.rb:55:in `bind'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/binding.rb:68:in `load_default_jars'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/binding.rb:68:in `each'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/binding.rb:71:in `block in load_default_jars'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/binding.rb:63:in `load_jar'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/jar_loader.rb:27:in `load'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/jar_loader.rb:38:in `load_jar_rjb'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/jar_loader.rb:45:in `init_rjb'
/Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/jar_loader.rb:45:in `load'

-- C level backtrace information -------------------------------------------

   See Crash Report log file under ~/Library/Logs/CrashReporter or
   /Library/Logs/CrashReporter, for the more detail of.

-- Other runtime information -----------------------------------------------

* Loaded script: irb

* Loaded features:

    0 enumerator.so
    1 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/enc/encdb.bundle
    2 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/enc/trans/transdb.bundle
    3 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems/defaults.rb
    4 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/rbconfig.rb
    5 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems/deprecate.rb
    6 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems/exceptions.rb
    7 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb
    8 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems.rb
    9 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/e2mmap.rb
   10 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/init.rb
   11 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/workspace.rb
   12 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/inspector.rb
   13 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/context.rb
   14 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/extend-command.rb
   15 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/output-method.rb
   16 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/notifier.rb
   17 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/slex.rb
   18 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/ruby-token.rb
   19 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/ruby-lex.rb
   20 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/src_encoding.rb
   21 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/magic-file.rb
   22 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/readline.bundle
   23 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/input-method.rb
   24 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/locale.rb
   25 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb.rb
   26 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/irb/completion.rb
   27 /Users/thomas/.rvm/scripts/irbrc.rb
   28 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems/version.rb
   29 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems/requirement.rb
   30 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems/platform.rb
   31 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb
   32 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems/path_support.rb
   33 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb
   34 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/version.rb
   35 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/exception.rb
   36 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/autoload.rb
   37 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/helpers/hash.rb
   38 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/helpers/help.rb
   39 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/helpers/object.rb
   40 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/helpers/string.rb
   41 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/config/configurable.rb
   42 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/config/importable.rb
   43 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/config/config.rb
   44 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/config/paths.rb
   45 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/config/tags.rb
   46 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/loaders/linguistics.rb
   47 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/loaders/stanford.rb
   48 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/learning/data_set.rb
   49 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/learning/export.rb
   50 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/learning/problem.rb
   51 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/learning/question.rb
   52 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/birch-0.0.7/lib/birch/native.bundle
   53 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/birch-0.0.7/lib/birch/version.rb
   54 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/birch-0.0.7/lib/birch.rb
   55 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/applicable.rb
   56 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/socket.bundle
   57 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/socket.rb
   58 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/timeout.rb
   59 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/net/protocol.rb
   60 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/uri/common.rb
   61 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/uri/generic.rb
   62 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/uri/ftp.rb
   63 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/uri/http.rb
   64 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/uri/https.rb
   65 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/uri/ldap.rb
   66 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/uri/ldaps.rb
   67 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/uri/mailto.rb
   68 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/uri.rb
   69 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/zlib.bundle
   70 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/stringio.bundle
   71 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/net/http.rb
   72 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/etc.bundle
   73 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/fileutils.rb
   74 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/delegate.rb
   75 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/thread.rb
   76 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/singleton.rb
   77 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/tmpdir.rb
   78 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/tempfile.rb
   79 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/dos_time.rb
   80 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/ioextras.rb
   81 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/zip_entry.rb
   82 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/zip_extra_field.rb
   83 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/zip_entry_set.rb
   84 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/zip_central_directory.rb
   85 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/zip_file.rb
   86 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/zip_input_stream.rb
   87 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/zip_output_stream.rb
   88 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/decompressor.rb
   89 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/compressor.rb
   90 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/null_decompressor.rb
   91 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/null_compressor.rb
   92 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/null_input_stream.rb
   93 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/pass_thru_compressor.rb
   94 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/pass_thru_decompressor.rb
   95 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/inflater.rb
   96 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/deflater.rb
   97 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/zip_streamable_stream.rb
   98 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/zip_streamable_directory.rb
   99 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/constants.rb
  100 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/settings.rb
  101 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rubyzip-0.9.9/lib/zip/zip.rb
  102 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/progressbar-0.12.0/lib/progressbar.rb
  103 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/schiphol-0.9.4/lib/schiphol/mimes.rb
  104 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/schiphol-0.9.4/lib/schiphol.rb
  105 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/buildable.rb
  106 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/checkable.rb
  107 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/comparable.rb
  108 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/countable.rb
  109 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/debuggable.rb
  110 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/delegatable.rb
  111 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/exportable.rb
  112 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/iterable.rb
  113 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/magical.rb
  114 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/registrable.rb
  115 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity/stringable.rb
  116 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entity.rb
  117 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/entities/entities.rb
  118 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/groupable.rb
  119 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/categorizable.rb
  120 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/workers.rb
  121 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/proxies/proxy.rb
  122 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/proxies/array.rb
  123 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/proxies/language.rb
  124 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/proxies/number.rb
  125 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/proxies/string.rb
  126 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/core/dsl.rb
  127 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/core/installer.rb
  128 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/core/server.rb
  129 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/modules.rb
  130 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/builder.rb
  131 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat.rb
  132 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/formatters/readers/autoselect.rb
  133 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/nokogiri.bundle
  134 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/version.rb
  135 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/syntax_error.rb
  136 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/pp/node.rb
  137 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/pp/character_data.rb
  138 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/pp.rb
  139 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/parse_options.rb
  140 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/sax/document.rb
  141 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/sax/parser_context.rb
  142 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/sax/parser.rb
  143 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/sax/push_parser.rb
  144 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/sax.rb
  145 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/node/save_options.rb
  146 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/node.rb
  147 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/attribute_decl.rb
  148 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/element_decl.rb
  149 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/element_content.rb
  150 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/character_data.rb
  151 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/namespace.rb
  152 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/attr.rb
  153 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/dtd.rb
  154 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/cdata.rb
  155 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/text.rb
  156 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/document.rb
  157 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/document_fragment.rb
  158 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/processing_instruction.rb
  159 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/node_set.rb
  160 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/syntax_error.rb
  161 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/xpath/syntax_error.rb
  162 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/xpath.rb
  163 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/xpath_context.rb
  164 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/builder.rb
  165 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/reader.rb
  166 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/notation.rb
  167 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/entity_decl.rb
  168 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/schema.rb
  169 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml/relax_ng.rb
  170 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xml.rb
  171 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xslt/stylesheet.rb
  172 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/xslt.rb
  173 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/html/entity_lookup.rb
  174 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/html/document.rb
  175 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/html/document_fragment.rb
  176 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/html/sax/parser_context.rb
  177 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/html/sax/parser.rb
  178 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/html/sax/push_parser.rb
  179 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/html/element_description.rb
  180 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/html/element_description_defaults.rb
  181 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/html.rb
  182 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/decorators/slop.rb
  183 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/css/node.rb
  184 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/css/xpath_visitor.rb
  185 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/racc/cparse.bundle
  186 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/racc/parser.rb
  187 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/css/parser_extras.rb
  188 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/css/parser.rb
  189 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/strscan.bundle
  190 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/css/tokenizer.rb
  191 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/css/syntax_error.rb
  192 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/css.rb
  193 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri/html/builder.rb
  194 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/nokogiri-1.5.6/lib/nokogiri.rb
  195 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/guess_html_encoding-0.0.9/lib/guess_html_encoding/version.rb
  196 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/guess_html_encoding-0.0.9/lib/guess_html_encoding.rb
  197 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/ruby-readability-0.5.6/lib/readability.rb
  198 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/ruby-readability-0.5.6/lib/ruby-readability.rb
  199 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/formatters/readers/html.rb
  200 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/enc/trans/single_byte.bundle
  201 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/digest.bundle
  202 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/digest.rb
  203 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/openssl.bundle
  204 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/openssl/bn.rb
  205 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/openssl/cipher.rb
  206 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/openssl/config.rb
  207 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/openssl/digest.rb
  208 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/openssl/buffering.rb
  209 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/fcntl.bundle
  210 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/openssl/ssl-internal.rb
  211 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/openssl/x509-internal.rb
  212 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/openssl.rb
  213 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/net/https.rb
  214 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/date_core.bundle
  215 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/date/format.rb
  216 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/date.rb
  217 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/time.rb
  218 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/open-uri.rb
  219 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/fastimage-1.2.13/lib/fastimage.rb
  220 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/processors/chunkers/autoselect.rb
  221 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/processors/chunkers/html.rb
  222 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/scalpel-0.2.1/lib/scalpel.rb
  223 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/processors/segmenters/scalpel.rb
  224 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/treat-2.0.3/lib/treat/workers/processors/parsers/stanford.rb
  225 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/stanford-core-nlp-0.5.1/lib/stanford-core-nlp/config.rb
  226 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rjb-1.4.5/lib/rjbcore.bundle
  227 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/rjb-1.4.5/lib/rjb.rb
  228 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/rjb_proxy.rb
  229 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/jar_loader.rb
  230 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it/binding.rb
  231 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/bind-it-0.2.6/lib/bind-it.rb
  232 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/stanford-core-nlp-0.5.1/lib/stanford-core-nlp/bridge.rb
  233 /Users/thomas/.rvm/gems/ruby-1.9.3-p362@mona/gems/stanford-core-nlp-0.5.1/lib/stanford-core-nlp.rb
  234 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/dl.bundle
  235 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/x86_64-darwin12.2.0/fiddle.bundle
  236 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/fiddle/function.rb
  237 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/fiddle/closure.rb
  238 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/fiddle.rb
  239 /Users/thomas/.rvm/rubies/ruby-1.9.3-p362/lib/ruby/1.9.1/dl.rb

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

Installer issue on JRuby / OSX

First off, thanks for all your hard work on treat!

I received the following error when trying to install treat in JRuby on OSX 10.9.

1.9.3 (main):0 > require 'treat'
=> true
1.9.3 (main):0 > Treat::Core::Installer.install('english')
LoadError: load error: jopenssl/load -- java.lang.ExceptionInInitializerError: null
from org/jruby/RubyKernel.java:1083:in `require'
1.9.3 (main):0 >

Looks like it's working in MRI 2.1.0 or at least getting farther and then running into native C extension issues... which is probably more of a local thing on my computer.

It looks like something in schiphol related to ssl isn't working under JRuby.

If I find a fix, I'll post back in this issue.

Treat::Exception: No TF_IDF is/are available for the English Language?

Hello,

First of all, thank you so much for creating this awesome gem. I have one issue while using this gem. I have trouble getting keywords out of collection. I just used the sample code you provided in the tutorial:

c = Collection('economist')
c.do(:chunk, :segment, :tokenize, :keywords)

I have installed tf_idf gem manually and it is in place. However, I still get this weird error:

Treat::Exception: No tf_idf is/are available for the English language.
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/delegatable.rb:112:in find_worker_for_language' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/delegatable.rb:85:infind_worker'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/delegatable.rb:57:in call_worker' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/delegatable.rb:42:inblock (2 levels) in add_workers'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/workers/extractors/keywords/tf_idf.rb:20:in block in keywords' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/magical.rb:45:inblock in magic'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/iterable.rb:22:in block (2 levels) in each_entity' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/iterable.rb:22:inblock (2 levels) in each_entity'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/iterable.rb:22:in block (2 levels) in each_entity' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/iterable.rb:18:ineach_entity'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/iterable.rb:22:in block in each_entity' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/iterable.rb:21:ineach'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/iterable.rb:21:in each_entity' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/iterable.rb:22:inblock in each_entity'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/iterable.rb:21:in each' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/iterable.rb:21:ineach_entity'
... 5 levels...
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/workers/extractors/keywords/tf_idf.rb:19:in keywords' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/delegatable.rb:68:incall_worker'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/delegatable.rb:42:in block (2 levels) in add_workers' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/doable.rb:39:indo_task'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/doable.rb:47:in block in do_task' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/doable.rb:46:ineach'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/doable.rb:46:in do_task' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/doable.rb:22:inblock in do'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/doable.rb:6:in each' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/treat-1.2.0/lib/treat/entities/abilities/doable.rb:6:indo'
from (irb):2
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.2.6/lib/rails/commands/console.rb:47:in start' from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.2.6/lib/rails/commands/console.rb:8:instart'
from /Users/kwonb/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.2.6/lib/rails/commands.rb:41:in <top (required)>' from script/rails:6:inrequire'
from script/rails

Sorry, I might be throwing a too basic question, but this problem cannot be solved by myself.. is there some additional steps I need to do?

bug french parser stanford

I did this simple script :

require 'treat'

Treat.default_language = :french
Treat.silence = false

s = 'Bonjour je suis bien au chateau'
s.parse

I have an error message :

Any help ?

Thanx !

Loading parser from serialized file /Users/ztera/.rvm/gems/ruby-1.9.2-p290/gems/treat-1.0.4/models/stanford/grammar/frenchFactored.ser.gz ...
java.lang.ClassNotFoundException: edu.stanford.nlp.parser.lexparser.FrenchUnknownWordModel

Loading parser from text file /Users/ztera/.rvm/gems/ruby-1.9.2-p290/gems/treat-1.0.4/models/stanford/grammar/frenchFactored.ser.gz /Users/ztera/.rvm/gems/ruby-1.9.2-p290/gems/stanford-core-nlp-0.3.0/lib/stanford-core-nlp.rb:124:in `new': /Users/ztera/.rvm/gems/ruby-1.9.2-p290/gems/treat-1.0.4/models/stanford/grammar/frenchFactored.ser.gz: expecting BEGIN block; got ¨Ì??sr??,edu.stanford.nlp.parser.lexparser.ParserData???????????????L??bgt??1Ledu/stanford/nlp/parser/lexparser/BinaryGrammar;L??dgt??5Ledu/stanford/nlp/parser/lexparser/DependencyGrammar;L??lext??+Ledu/stanford/nlp/parser/lexparser/Lexicon;L??ptt??+Ledu/stanford/nlp/parser/lexparser/Options;L?? (RuntimeException)

is there a limit on the number of words that can be in a sentence that treat can parse? the text i'm trying to work with has really long sentences, often over 100 words per.

Installer is doing too many things

I like the automated installer but it appears to be duplicating the efforts of rubygems.

For instance in dependencies.rb there are listed 12 gems which could simply be listed as runtime dependencies in the treat.gemspec file.

Also, the external binary libraries doesn't inform you of what to install if you choose to install manually. That information is only available on the wiki.

This is more of meta issues, but I did find a small bug as well which I will fork and fix.

Parsing problem

hi, i try to install this gem and i receive the next errror:

C:\Users\username>gem install treat --version 2.0.7
Thanks for installing Treat.
To complete the installation, run require treat in an IRB
terminal, followed by Treat::Core::Installer.install.
Successfully installed treat-2.0.7
Parsing documentation for treat-2.0.7

RDoc::Parser::Ruby failure around line 214 of
lib/treat/entities/entity/buildable.rb

Before reporting this, could you check that the file you're documenting
has proper syntax:

C:/Ruby200-x64/bin/ruby.exe -c lib/treat/entities/entity/buildable.rb

RDoc is not a full Ruby parser and will fail when fed invalid ruby programs.

The internal error was:

    (NoMethodError) undefined method `name' for {TkNL 5521, 213:40 "\n"}:RDo

c::RubyToken::TkNL

ERROR: While executing gem ... (NoMethodError)
undefined method `name' for {TkNL 5521, 213:40 "\n"}:RDoc::RubyToken::TkNL

C:\Users\username>

thanks for help me

LDA with Russian

Hi there!
I am curious if there is any opportunity to build support of Russian language inside LDA package. Is there any easy way to do this?

1.2 - JAVA Wordnet interface.

The goal for this milestone is to provide a fast JAVA Wordnet interface in addition to the relatively slow pure Ruby interface that is already available. Most probably, the best library to interface would be JWI (http://projects.csail.mit.edu/jwi/).

Manual install of zip gem needed to complete install

Needed to manually install zip gem to complete install via irb.

Mac OS X 10.8
rvm 1.22.13
ruby-1.9.3-p392 [ x86_64 ]
ruby-2.0.0-p247 [ x86_64 ]

Documentation on retrieving from Mongo

I'm back at it and would like to contribute to the documentation again.

I've used the chunker (html) and segmenter (stanford) to break down an HTML document into sentences. I am storing the parsed document now in Mongo using the serialized, but I can't seem to figure out the API for retrieving one and the same.

This is more of a users group question, which I'd be happy to post on, but I would like to add the docs myself when I figure this out.

tf*idf: Why don't you normalize to maximum term count for document?

Treat (and the tf-idf and similarity gems) all normalize tf to the number of terms in the document: https://github.com/louismullie/treat/blob/master/lib/treat/workers/extractors/tf_idf/native.rb#L78

We normalize so that (1) long and short documents have comparable tf weights and (2) documents with large vocabularies and those with small vocabularies have comparable tf weights.

Normalizing to the number of terms in a document only fixes (1). Normalizing to the maximum term count (which is all I've ever seen in the literature) fixes both (1) and (2).

Treat should not throw any warnings in verbose mode

German installation fails

Installing stanford... Gem::GemNotFoundException: Could not find a valid gem 'stanford' (>= 0) locally or in a repository from /Users/jack/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/dependency_installer.rb:237:in `find_spec_by_name_and_version'

Perhaps this should be stanford-nlp?

Stanford parser and named entity tagger should accept tokenized sentences

Currently, they only accept untokenized sentences and raise a warning if a tokenized sentence is provided.

www.louismullie.com is not available

$ ping -c 5 www.louismullie.com
PING www.louismullie.com (216.70.82.70): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3

--- www.louismullie.com ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

So I can't download dependencies for my language.

invalid multibyte char (US-ASCII)

I'm getting this error below, after requiring treat in my irb console

1.9.3-p0 :001 > require 'treat'
SyntaxError: (eval):60: invalid multibyte char (US-ASCII)
(eval):60: invalid multibyte char (US-ASCII)
(eval):60: syntax error, unexpected $end, expecting ']'
      "für",
          ^

What is the problem here,

Environment:
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-darwin11.0.0]
treat (2.0.3)

Thanks

Information extraction

Thanks for the "Treat". I think the gem is the most comprehensive gem for NLP by far!

I'm trying to extract the keywords or topics given a sentence or paragraph. I read the "Topic Word Extraction" and "General Topic Extraction" in the link (https://github.com/louismullie/treat/wiki/Information-Extraction) and that's what I am looking for...

Questions:

Is there a way to perform "Topic Word Extraction" from a Paragraph class. It looks like "topic_words" supports Collection class only.

For "General Topic Extraction", is there a way to get the list of the topic words after doing the "topics" method? The example shows how to export it to a DOT file and and I see some other examples doing "print_tree". I am wondering if we can do something like to_a ?

Thanks!

Is this the kind of output I can expect?

I went through the installation and I'm trying this out ... but I'm not seeing that the output is that useful.

Here is my test run:

=> nil
1.9.3p194 :002 > "I am a sentence".parse
=> Phrase (28923900) --- "I am a sentence" --- {:tag=>"S", :tag_set=>:penn} --- []
1.9.3p194 :003 > "This should not be hard".parse
=> Phrase (31930400) --- "This should not be hard" --- {:tag=>"S", :tag_set=>:penn} --- []
1.9.3p194 :004 > "I would like to go to the movies".parse.inspect
=> "Phrase (31966680) --- "I would like [...] the movies" --- {:tag=>"S", :tag_set=>:penn} --- [] "
1.9.3p194 :005 > "I would like to go to the movies".tokenize
=> Phrase (26454480) --- "I would like [...] the movies" --- {} --- []
1.9.3p194 :006 > "This is a sentence.".tokenize
=> Sentence (26860740) --- "This is a sentence." --- {} --- []
1.9.3p194 :007 > sect = Section "A walk in the park\n"+
'Obama and Sarkozy met this friday to investigate ' +
'the possibility of a new rescue plan. The French ' +
'president Sarkozy is to meet Merkel next Tuesday.'

sect.do :chunk, :segment, :tokenize, :parse1.9.3p194 :008 > 1.9.3p194 :009 > 1.9.3p194 :010 > => Section (25993960) --- "A walk in [...] next Tuesday." --- {} --- []
1.9.3p194 :011 > 1.9.3p194 :012 > p = Paragraph 'A walk in the park. A trip on a boat.'
p.segmentSyntaxError: (irb):12: syntax error, unexpected '=', expecting $end
... :segment, :tokenize, :parsep = Paragraph 'A walk in the par...
... ^
from /home/dave/.rvm/rubies/ruby-1.9.3-p194/bin/irb:16:in <main>' 1.9.3p194 :013 > p = Paragraph 'A walk in the park. A trip on a boat.' NoMethodError: undefined methodsegmentp=' for nil:NilClass
from (irb):13
from /home/dave/.rvm/rubies/ruby-1.9.3-p194/bin/irb:16:in `

'
1.9.3p194 :014 > s = "How do you think this works"
=> "How do you think this works"
1.9.3p194 :015 > s.tokenize
=> Phrase (25719320) --- "How do you think this works" --- {} --- []
1.9.3p194 :016 > s = Sentence "This is a sentence."
=> Sentence (24476840) --- "This is a sentence." --- {} --- []
1.9.3p194 :017 > s.tokenize
=> Sentence (24476840) --- "This is a sentence." --- {} --- []
1.9.3p194 :018 >

Is this typical output? Should I be seeing more info on this stuff?

Thanks

Rails

Do you have any documentation to get this into a rails app? Thanks!

XML Serialize ":" punctuation bug

Colon punctuation in XML file like this
<punctuation id='2157138080' tag=':' >;</punctuation>

Throws this error
/Users/zkt/.rvm/gems/ruby-1.9.3-p327/gems/treat-2.0.4/lib/treat/core/dsl.rb:17:in method_missing': undefined methodto_i' for :"":Symbol (NoMethodError)
from /Users/zkt/.rvm/gems/ruby-1.9.3-p327/gems/treat-2.0.4/lib/treat/workers/formatters/unserializers/xml.rb:68:in block in unserialize' from /Users/zkt/.rvm/gems/ruby-1.9.3-p327/gems/treat-2.0.4/lib/treat/workers/formatters/unserializers/xml.rb:40:ineach_pair'
from /Users/zkt/.rvm/gems/ruby-1.9.3-p327/gems/treat-2.0.4/lib/treat/workers/formatters/unserializers/xml.rb:40:in unserialize' from /Users/zkt/.rvm/gems/ruby-1.9.3-p327/gems/treat-2.0.4/lib/treat/entities/entity/delegatable.rb:66:incall_worker'
from /Users/zkt/.rvm/gems/ruby-1.9.3-p327/gems/treat-2.0.4/lib/treat/entities/entity/delegatable.rb:40:in block (2 levels) in add_workers' from /Users/zkt/.rvm/gems/ruby-1.9.3-p327/gems/treat-2.0.4/lib/treat/entities/entity/buildable.rb:264:infrom_serialized_file'
from /Users/zkt/.rvm/gems/ruby-1.9.3-p327/gems/treat-2.0.4/lib/treat/entities/entity/buildable.rb:213:in from_file' from /Users/zkt/.rvm/gems/ruby-1.9.3-p327/gems/treat-2.0.4/lib/treat/entities/entity/buildable.rb:55:inbuild'
from /Users/zkt/.rvm/gems/ruby-1.9.3-p327/gems/treat-2.0.4/lib/treat/core/dsl.rb:12:in method_missing' from lib/pre_test.rb:208:inblock in

'
from lib/pre_test.rb:205:in each' from lib/pre_test.rb:205:in

In the workers/formaters/unserializers/xml.rb file is the following code (lines 67 and 68) which causes the error it produces an empty symbol when the colon is processed which then matches the to_i regex

 67     v = v[1..-1].intern if v[0] == ':'
 68      v = v.to_i if v =~ /^[0-9]*$/

TypeError: no implicit conversion from nil to integer when tokenizing

Running through a few examples from the manual, cannot get tokenize to work. Here is a paste from irb:

2.0.0p195 :005 > require 'treat'
 => false
2.0.0p195 :006 > include Treat::Core::DSL
 => Object
2.0.0p195 :007 > p = paragraph('A walk in the park. A trip on a boat.').segment
 => Paragraph (70247709531120)  --- "A walk in [...] a boat."  ---  {}   --- []
2.0.0p195 :023 > Treat.core.language.detect = true
 => true
2.0.0p195 :035 > sent = "This is an English sentence, prove it to me!"
 => "This is an English sentence, prove it to me!"
2.0.0p195 :036 > sent.apply(:tokenize, :parse).print_tree
TypeError: no implicit conversion from nil to integer
    from /Users/mac/.rvm/gems/ruby-2.0.0-p195/gems/treat-2.0.7/lib/treat/entities/entity/delegatable.rb:91:in `[]'
    from /Users/mac/.rvm/gems/ruby-2.0.0-p195/gems/treat-2.0.7/lib/treat/entities/entity/delegatable.rb:91:in `find_worker_for_language'
    from /Users/mac/.rvm/gems/ruby-2.0.0-p195/gems/treat-2.0.7/lib/treat/entities/entity/delegatable.rb:83:in `find_worker'
    from /Users/mac/.rvm/gems/ruby-2.0.0-p195/gems/treat-2.0.7/lib/treat/entities/entity/delegatable.rb:55:in `call_worker'
    from /Users/mac/.rvm/gems/ruby-2.0.0-p195/gems/treat-2.0.7/lib/treat/entities/entity/delegatable.rb:40:in `block (2 levels) in add_workers'
    from /Users/mac/.rvm/gems/ruby-2.0.0-p195/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:41:in `do_task'
    from /Users/mac/.rvm/gems/ruby-2.0.0-p195/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:22:in `block in apply'
    from /Users/mac/.rvm/gems/ruby-2.0.0-p195/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:6:in `each'
    from /Users/mac/.rvm/gems/ruby-2.0.0-p195/gems/treat-2.0.7/lib/treat/entities/entity/applicable.rb:6:in `apply'
    from /Users/mac/.rvm/gems/ruby-2.0.0-p195/gems/treat-2.0.7/lib/treat/proxies/proxy.rb:14:in `method_missing'
    from (irb):36
    from /Users/mac/.rvm/rubies/ruby-2.0.0-p195/bin/irb:16:in `<main>'

Missed file during installation

I'm runnint Treat.install and select default optiions (1)
And got next:

Treat::Exception: Couldn't download www.louismullie.com/treat/punkt/english.yaml. (response code was not 200 OK, but was 404. )

Support for better dependency management.

Why is gem dependency resolved with "Treat::Core::Installer.install 'english' ", couldn't this be a part of gemspec.
If support for multiple languages requires additional gems then we can just make it modular by creating new gem called 'treat-german'. one can install this if he needs support for german lang. I think Using this method of manually installing dependency will not take full advantage of 'bundler'.

Would like to know if there was any specific reason why dependency resolved manually.

proposal:

The base gem has all the 'agnostic' dependency as a part of gemspec.
Any language specific dependency/lib can be pulled out to a new gem with name "treat-"
This pattern can be observed in https://github.com/intridea/omniauth gem.

this can be much easy to maintain and will support bundler fully.

louismullie / treat Goto Github PK

treat's Introduction

treat's People

Contributors

Stargazers

Watchers

Forkers

treat's Issues

C:\Users\username>

Recommend Projects

Recommend Topics

Recommend Org