Arachnado is a tool to crawl a specific website. It provides a Tornado-based HTTP API and a web UI for a Scrapy-based crawler.
License is MIT.
Python 2.7 is required to run server. To build static assets node.js + npm are required.
Install all Python requirements from requirements.txt using pip:
pip install -U -r requirements.txt
Install all JavaScript requirements using npm. Run the following command from the repo root:
npm install
then rebuild static files (we use Webpack):
npm run build
or auto-build static files on each change during development:
npm run watch
To run Arachnado execute the following command from the repo root:
python -m arachnado
Then visit http://127.0.0.1:8888/