apokryptein / scraping Goto Github PK

JMO but I'd wrap this guy in a try block:

def storeUrl(url):
	cursor.execute("INSERT INTO pages (content) VALUES (\"%s\")", (url))
	cursor.connection.commit()

There's so much that can go wrong on the insert (PK error, lost connection, tablespace full, etc) that you may want to decide how you want to handle a failed insert. Outside of outright killing the program, you could report the error to the user and keep going with an f.open() to write everything to a flat file so you're still recording.