fizx / robots Goto Github PK

This is what I do:
ruby-head > bla = Robots.new("test")
=> #<Robots:0x000001008743c0 @user_agent="test", @parsed={}>
ruby-head > bla.allowed?("http://lacostarecords.net/")
=> false

This is the robots.txt
# Block a bot that was causing issues by ignoring Disallow lines below
User-Agent: OmniExplorer_Bot
Disallow: /

# Block hotlinking of music files by projectplaylist.com due to perceived user bandwidth theft
User-agent: projectplaylist-directlink
Disallow: /

# Block all bots from the core Homestead site
User-agent: *
Disallow: /~site/Scripts_ElementMailer
Disallow: /~site/Scripts_ExternalRedirect
Disallow: /~site/Scripts_ForSale
Disallow: /~site/Scripts_HitCounter
Disallow: /~site/Scripts_NewGuest
Disallow: /~site/Scripts_RealTracker
Disallow: /~site/Scripts_Track
Disallow: /~site/Scripts_WebPoll

it should return true, shouldn't it?

This doesn't seem to work:

User-agent: *
Disallow:

(found here: http://www.gelsenclan.de/robots.txt )
Having a robots.txt like this will result in NO part of the page being allowed.
The expected behavior would be that everything is allowed

Getting 'LoadError: no such file to load -- robots' when I try to require 'robots' in irb. Have robots 0.10.1 installed.

Hello ,
please would you consider including patch to the Rakefile as suggested by Mamoru Tasaka?
https://bugzilla.redhat.com/show_bug.cgi?id=632912

$ diff -u Rakefile.debug Rakefile
--- Rakefile.debug 1970-01-01 09:00:00.000000000 +0900
+++ Rakefile 2010-09-13 02:32:52.000000000 +0900
@@ -21,7 +21,7 @@
require 'rake/testtask'
Rake::TestTask.new(:test) do |test|
test.libs << 'lib' << 'test'

• test.pattern = 'test/*_/__test.rb'
• test.pattern = 'test/*_/test__.rb'
  test.verbose = true
  end

@@ -38,7 +38,7 @@
end
end

-task :test => :check_dependencies
+#task :test => :check_dependencies

task :default => :test

It would be nice if the "robots.txt request timed out" would tell for which uri it actually failed.
This is the method in question:

  def self.get_robots_txt(uri, user_agent)
    begin
      Timeout::timeout(Robots.timeout) do
        io = URI.join(uri.to_s, "/robots.txt").open("User-Agent" => user_agent) rescue nil
      end 
    rescue Timeout::Error
      STDERR.puts "robots.txt request timed out"
    end
  end

I noticed you request /robots.txt using open-uri. Since open-uri overrides Kernel#open, it also allows for possible command injection (ex: open("| ls")). I think using Net::HTTP.get would be Good Enough (tm).

System: ruby-head and 1.9.1, using the newest gem version.
I came across the error over here: http://danielwebb.us/robots.txt

User-agent: *
Disallow: /bot-trap
Disallow: /about/contact
Disallow: /about/resume/daniel_webb-resume.pdf
Disallow: /projects/pd_tech_books/the_boy_electrician.pdf

But still:

ruby-head > robots.allowed?("http://danielwebb.us/bot-trap/index.php")
 => true 
ruby-head > robots.allowed?("http://danielwebb.us/bot-trap/")
 => true 
ruby-head > robots.allowed?("http://danielwebb.us/bot-trap")
 => true 

What is even more disturbing:
ruby-head > robots.allowed?("http://www.google.de/search")
=> true

Did something break in the recent update?
Did something break in one of the last updates?