how can i do multiple pdf extraction processes concurrently? about docsplit HOT 5 CLOSED

We are using a redis queue with circus to lauch X workers. And it works fine.

http://redis.io/
http://circus.readthedocs.org/

from docsplit.

Hey @quyen can you be a little more specific about what errors you're encountering?

DocumentCloud uses Docsplit in a manner similar to what @Natim outlines.

from docsplit.

@knowtheory & @Natim - I'm trying to do the same thing as Quyen, but having some trouble figuring out Circus..

Would you guys happen to know of any tutorial covering the set-up for using Circus to run multiple processes?

Thanks in advance.

from docsplit.

@antonlakin Have a look at thoose projects : https://github.com/novagile/insight-reloaded and https://github.com/novagile/insight-installer there is some configuration example : https://github.com/novagile/insight-installer/tree/master/chef/cookbooks/insight/templates/default

from docsplit.

Just for some additional details, DocumentCloud uses CloudCrowd for distributed queuing of jobs which use DocSplit. You can check out the actions we've written, and in particular note the document_import action.

from docsplit.