It'd be awesome if you could use that when available, because in my limited tests, it runs faster, even with rb-gsl installed.

b = Classifier::Bayes.new "yes", "no"
b.train_yes "yes"
b.train_yes "ok"

b.train_no "no"
b.train_no "nope"
b.train_no "stop"
b.classify "ok"

=> "No"

It looks like the prior probability for each classification is ignored in the Naive Bayes classifier. You should keep track of the number of times "train" is called with each class, and then use that to calculate the prior for each class: P(class) = # class trainings / # all trainings. You should then add Math.log(P(class)) to your log sum (score[category.to_s]) in your "classifications" method.

Without the addition of that prior you do not have a Naive Bayes classifier.

Hi,

I noticed that 1.3.5 gem version doesn't include the tests, whereas they were present in the 1.3.4 version. Could you please continue to include them? For Debian packages, we try to use the gem as the base of source package, and it is useful for our quality assurance that tests are present directly there.

Thanks a lot

Cedric

We included your gem to http://gems.sciruby.com/ and we use the source information from github for synchronization. Just log in at rubygems.org, edit the gem and add a source link. Thx!

Hi,

Could you please tag commits corrsponding to releases in the repositories? The last gem version is 1.3.5 but it is not easy to spot which commit it corresponds to. This would allow also automatic creation of tarballs from the tag, useful for packagers for distributions.

Thanks.

Cédric

There is a lot of info about the quality of a classification guess
in LSI which is hidden from the caller.

Sometimes, the guess is of very high quality and sometimes it's a
toss-up. The caller has no way to distinguish them though.

It would be nice if there was an option to get not only the classification
but a measure of the system's confidence.

I propose adding a method classify_with_confidence(text) which returns
both the classification AND a measure of the confidence as a number
between 0 and 1.0

i.e. guess, confidence = lsi.classify_with_confidence(text)

Then a caller could choose not to use any classification guesses whose
confidence is below some threshold.

I see a method to add a category from the bayesian filter... but is there a way to remove one?

Unfortunately the problem I was having before still isn't solved...here's the error I'm getting now: /Library/Ruby/Gems/1.8/gems/classifier-1.3.3/lib/classifier/extensions/vector.rb:16:insum': undefined method to_f' for []:Array (NoMethodError)

I've noticed the LSI classifier fails on certain input strings. It appears to be related to repeating words, but I haven't figured out the exact pattern. As an example:

require 'classifier'
lsi = Classifier::LSI.new
strings = [
            ["This text deals with dogs. dogs.", "dog"],
            ["This text involves dogs too. Dogs! ", "dog"],
            ["Remind me to get milk on monday", "reminder"]
          ]

strings.each {|x| lsi.add_item x.first, x.last}

puts lsi.classify "Remind me to Remind"

results in

/Users/willstrinz/.rvm/gems/ruby-2.0.0-p353/gems/classifier-1.3.4/lib/classifier/lsi.rb:190:in `sort_by': comparison of Float with Float failed (ArgumentError)
    from /Users/willstrinz/.rvm/gems/ruby-2.0.0-p353/gems/classifier-1.3.4/lib/classifier/lsi.rb:190:in `proximity_array_for_content'
    from /Users/willstrinz/.rvm/gems/ruby-2.0.0-p353/gems/classifier-1.3.4/lib/classifier/lsi.rb:255:in `classify'
    from lsi.rb:11:in `<main>'

but if I change the repeat of "Remind" it works fine"

puts lsi.classify "Remind me to zzRemind"
 #=> remind

It looks like the vectorization process is somehow getting NaNs in the ContentNode creation process, which is the direct source of the error, but I haven't been able to track down what's causing that yet.

There's a consistently unacceptable behavior of choosing the first category as default when it cannot determine the proper category.

Will confirm later with examples.

I think that's a good idea to make a function that let us to save the current state of training and able to continue training with new content in the future.
I know that we can use marshaling, but it would be nice to make a method 'save'.

require 'mathn' breaks calculations:

(10 / 23).class
=> Fixnum
require 'classifier'
(10 / 23).class
=> Rational

This means e.g. date calculations will break:

Date.civil(year, month, day)
=> ArgumentError: invalid date 

See:
http://blog.brightredglow.com/2008/1/17/evil-can-be-dangerous/

I'm trying to create a classifier using the cardmagic classifier gem. This is my code:

require 'classifier'

classifications = '1007.09', '1006.03'
traindata = Hash["1007.09" => "ADAPTER- SCREENING FOR VALVES VBS", "1006.03" => "ACTUATOR- LINEAR"]

b = Classifier::Bayes.new classifications

traindata.each do |key, value|
b.train(key, value)
end

But when I run this i get the following error:

Notice: for 10x faster LSI support, please install http://rb-gsl.rubyforge.org/
c:/Ruby192/lib/ruby/gems/1.9.1/gems/classifier-1.3.3/lib/classifier/bayes.rb:27:in block in train': undefined method[]' for nil:NilClass (NoMethodError)
from c:/Ruby192/lib/ruby/gems/1.9.1/gems/classifier-1.3.3/lib/classifier/bayes.rb:26:in each' from c:/Ruby192/lib/ruby/gems/1.9.1/gems/classifier-1.3.3/lib/classifier/bayes.rb:26:intrain'
from C:/_Chris/Code/classifier/smdclasser.rb:13:in block in <main>' from C:/_Chris/Code/classifier/smdclasser.rb:11:ineach'
from C:/_Chris/Code/classifier/smdclasser.rb:11:in `

This is the source from the gem code:

Provides a general training method for all categories specified in Bayes#new
For example:
b = Classifier::Bayes.new 'This', 'That', 'the_other'
b.train :this, "This text"
b.train "that", "That text"
b.train "The other", "The other text"
def train(category, text)
category = category.prepare_category_name
text.word_hash.each do |word, count|
@categories[category][word] ||= 0
@categories[category][word] += count
@total_words += count
end
end

I am lost where to go to troubleshoot this error, what is the next step i should take?

I'm using Jekyll, which uses the Classifier gem to index related posts. I have GSL 1.14 installed via Homebrew, and it's in my path. I have the GSL gem 1.14.7 installed (using rbenv, but I can load it in irb so that's not the problem). When Classifier runs, it raises this error:

Notice: for 10x faster LSI support, please install http://rb-gsl.rubyforge.org/

What should I try to get Classifier to recognize the GSL gem?

If anyone is interested in continuing work on this project, we've forked it over at https://github.com/jekyll/classifier-reborn. We just released a "v2.0.0" which contains many of the fixes addressed in the PR's and issues here.

I am using sourceclassifier gem, but whenever i want to identify a piece of code an error pops out. From the stacktrace I understood that this happens in classifier 'classification' method.

Here is the full stacktrace:

from /Users/olegkiviljov/.rvm/gems/ruby-2.0.0-p247/gems/classifier-1.3.4/lib/classifier/bayes.rb:88:in classify' from /Users/olegkiviljov/.rvm/gems/ruby-2.0.0-p247/gems/chrislo-sourceclassifier-0.2.3/lib/sourceclassifier.rb:19:inidentify'
from experiment.rb:21:in `

I posted the same issue in sourceclassifier issues but with no avail. If someone could atleast tell me if it is a problem with sourceclassifier or with classifier, this would give me enough information to try and fix it myself.

Using active_record 2.3.8 and classifier 1.3.1, it appears that requiring classifier causes an error with active_record.

You can see what I'm talking about (hopefully) by running the test Ruby script at http://gist.github.com/463836 (and creating a test database with the included mysql dump). Works fine if you don't require classifier, errors if you do.

The trace is also included in the gist.