6Spider is a python based crawler, used to crawl specified job postings from job posting sites.
- Python 2.7.9
-
create a virtual python environment
pip install virtualenv
python -m virtualenv .virtualenv
-
install dependencies
pip install -r requirements.txt
-
test
py.test ./tests
-
execute
scrapy crawl weban