This project forked from f-debug/newsspider
该项目是基于Scrapy框架的Python新闻爬虫，能够爬取网易，搜狐，凤凰和澎湃网站上的新闻，将标题，内容，评论，时间等内容整理并保存到本地
该项目是基于Scrapy框架的Python新闻爬虫，能够爬取网易，搜狐，凤凰和澎湃网站上的新闻，将标题，内容，评论，时间等内容整理并保存到本地
项目需求
1：爬取网易，搜狐，凤凰和澎湃新闻网站的文章及评论
2：新闻网页数目不少于10万页
3：每个新闻网页及其评论能在1天内更新
项目技术
1:设计一个网络爬虫，能够爬取指定网站的全部页面，并提取其中的文章及评论内容
2:定时运行网络爬虫，实现每日更新数据
首先从初始URL 开始，Scheduler 会将其交给 Downloader 进行下载，下载之后会交给 Spider 进行分析，这里的spider就是爬虫的核心功能代码,Spider分析出来的结果有两种：一种是需要进一步抓取的链接，它们会通过middleware传回 Scheduler ；另一种是需要保存的数据，送入Item Pipeline ，进行处理和存储,最后将所有数据输出并保存为文件
项目结构
scrapyspier为scrapy框架文件
spiders是核心文件，对新闻网站信息爬取
debug_xxx.py是运行文件
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.
