python3 scraper.py <file of craigslist search URLs> [<email to notify>]

http://kaufman.mit.edu/projects/craigslist_bot.html

This script automates Craigslist searching by finding unique search results and reporting them over email. My specific use case is in apartment hunting: to find an apartment you need to rerun the same searches frequently so you can find that great new listing before anyone else. But in the process you spend a lot of time rereading old listings or reading repostings of old listings. The goal of this script is to eliminate all the rereading.

• The bot requires a SMTP server to send email.
• I didn't want the numerous September listings, so I filtered them out. You may want to remove that filter.

python3 scraper.py <file_of_search_urls> [<email_to_notify>]

• file_of_search_urls should be a file with one URL per line. To create this file, run your Craigslist search in the browser and save the URL that gives you back a list of listings.
• email_to_notify is an email to send unique results to. The bot gives fairly verbose output on the command line -- you probably want the digest by email.
• craig.db is a file generated by the script used to record the listings it has already seen in previous searches.

• The script only parses the first page of results. There's no real reason why it can't do more, but none of my searches required it so I didn't bother to implement that feature.
• If a listing is reposted with a new title and any different images the bot will think it is unique. Again, there's no fundamental limitation here, but I found the simple uniqueness criteria described above to be sufficiently effective that it wasn't worth coding something more complicated
• Email notifications assume access to MIT's SMTP server. If you have some other SMTP server you'll have to change that in the code. If you don't have a SMTP server then email notifications will not work for you.
• I decided to filter out September listings because there were I lot of them and I wasn't looking for September. If you want September listings then this is a limitation, not a feature.