Comments (1)
oa414
commented on June 20, 2024
1
你好， Scrapy 的文档 上写着，最新版本支持Python2.7 和 Python 3.3+(Windows 下不支持 Python 3)。
如果你用是 Windows 系统，我建议你尝试安装 Python 2.7 来试试。你可以用 Virtualenv 方便地来在系统中安装管理多个 Python 版本。
LinkExtractor 是 Scrapy 内建的一个 Extractor，它的作用就是从页面中分析所有链接，符合这个 LinkExtractor 规则的都会给你抽取出来，继续爬取。
自己手写爬虫遍历的思路是把当前已经处理过的页面 url 保存到数据库或者磁盘，中断之后重新爬取的时候，从保存的状态恢复而不是从头再来。 简单的思路是把当前的程序的数据定时序列化（比如用 pickle）到一个文件中，启动的时候先读取磁盘的状态来初始化 url。
当然 Scrapy 背后帮你做了很多这样的工作，比如从页面中抽取 url，url 有没有重复，并发爬取的时候的状态共享，都帮你解决了。
from appcrawler.
Related Issues (2)
- 有个问题 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from appcrawler.