A web spider for zhihu.com, which is used for ZhihuHot.
This spider can scrape question & topic data from zhihu.com.

I'm Morgan Zhang, a graduate computer science student in University of San Francisco.

See my another command line tool line here, which can tell you the amount of lines and files under current directory.

Coding for fun as well as learn Python.

[email protected]

• Python 2.7.6 (Maybe it work for other versions.)
• MySQL
• BeautifulSoup

1. Download the code
2. Run init.sql to set up datebase and insert initial data.
3. Find out your cookie of zhihu.com throught browser's developer tool.
4. Modify config.ini
5. Use python question.py to scrpy questions from zhihu.com
6. Use python topic.py to scrpy topics from zhihu.com

You can change thread amount in config.ini to make this spider run faster.
But your IP may be blocked from zhihu.com if you connect to zhihu.com too frequently.
You'd better use proxy when you use multi thread mode.

The MIT license.