Crawls all the urls on a specified domain and creates an entry for each in the Django database.

Create the database: python manage.py syncdb

The location of the database file is hardcoded in the settings file. I couldn't figure out a way around this so it will have to be modified. Edit website/settings.py and modify the Name field under Databases to reflect the absolute path of the database.

Set environment variables: . ./setup

Start RabbitMQ: sudo rabbitmq-server

Start Celery: cd $SCRAPER_HOME/scraper; celery -A tasks worker --loglevel=info

Run the scraper: cd $SCRAPER_HOME; python manage.py crawl <url>

Start the admin interface: cd $SCRAPER_HOME; python manage.py runserver

View the admin interface: http://localhost:8000/admin

Stop Celery Tasks: celeryctl purge

Stop RabbitMQ: sudo rabbitmqctl stop

Run test website: cd $SCRAPER_HOME/test; python testsite.py

Run scraper against test website: cd $SCRAPER_HOME; python manage.py crawl http://localhost:8080/page1