crawler
is a simple webcrawler that crawls all links on the subdomain of a given starting URL.
- For each page that is crawled,
crawler
will output the visited page URL and a list of links found on the page. - Links on the same subdomain as the starting URL that have not been seen before will then be crawled and output in the same way.
- Links on a different subdomain or an external domain are still printed, but not followed.
- Links to non-HTML resources such as PDF documents or media files are still printed but not followed.
crawler
respectsdisallow
/nofollow
directives from the siterobots.txt
file, as well asx-robots-tag
or<meta name="robots">
directives from individual pages.
pip install crawler
crawler <root_domain> [--num-workers N] [--timeout T]
Use crawler --help
for more information on optional parameters.
To create an editable install with all required development dependencies, run the following commands:
pip install -r requirements-dev.txt
pip install -e .
Installing in a python virutal environment for development is strongly recommended.
crawler
uses the pytest
framework for unit testing. All tests are located in the tests/
subdirectory, and can be run with:
pytest tests
crawler
follows PEP8 coding style, linting can be performed by running:
pylint crawler tests