dideler / caribou-data-collection Goto Github PK

Will easily indicate how old the crawled data is.

Because it's awesome! https://github.com/docopt/docopt

Haven't decided on the design yet. It can either import all scrapers or selectively import scrapers it needs.

Still need to figure out if all scrapers will share the same argument flags and how to determine which flags to use from main script.

Currently the .csv data is being written in the same directory as the module, and the unique data is being written to the top-level directory. Both of these should go to the data directory.

Specifically the methods that belong to the Crawler class and have to be executed in order.

The source contains a little duplicate data. Add an option so the user can easily remove duplicates when running the crawler. Neater to do it as part of the script than from the terminal as post-processing.