lhcardoso / crawwwler Goto Github PK

In order to run crawwwler in a clustered environment, we need network
wrapper classes associated with each and every plugin.  This one for
CCrawlerPlugin can be the protoype.

To illustrate: If e.g we want to run the crawler plugin on one server and
the url extractor plugin on another, then we would need the following:
Server 1:
   CUrlExtractor
   CCrawlerPluginNetworkAdapter   (verbose name for explanation)

Server 2:
  CCrawlerPlugin

The CUrlExtractor would subscribe to the CCrawlerPluginNetworkAdapter which
would act as a dummy.  It would pass data between the CUrlExtractor and the
CCrawlerPlugin on server 2.  Nothing more.

It is possible that this functionality could be built in to each of the
plugins and toggled by means of a switch.  I'm really not too sure how this
aspect will work yet, so I'm open for discussion.

A web front-end to search the database. Probably Ruby on Rails

Need to design and implement the main back end data store now that a basic
crawling prototype has been implemented

Need to implement a plugin that inherits from CPlugin that will connect to
the sql data store.

Initially this plugin needs only to subscribe to the CCrawlerPlugin and
save data about the server along with the raw html data to the data store