Python-based web scraper for phpBB forums. Project can be used as a template for building your own custom Scrapy spiders or for one-off crawls on designated forums. Please keep in mind that aggressive crawls can contribute significant strain on web servers, so please throttle your request rates.
- Python web scraping library, Scrapy.
- Python HTML/XML parsing library, BeautifulSoup.
The phpBB.py spider scrapes the following information from forum posts:
- Username
- User Post Count
- Post Date & Time
- Post Text
- Quoted Text
If you need additional data scraped, you will have to create additional spiders or edit the existing spider.
allowed_domains
start_urls
username
&password
forum_login=False
orforum_login=True
cd phpBB_scraper/
scrapy crawl phpBB
# scrapy crawl phpBB -o posts.csv
NOTE: Please adjust settings.py
to throttle your requests.