2018-12-04 11:05:45 [scrapy] INFO: Scrapy 1.0.3 started (bot: tumblrSpider)
2018-12-04 11:05:45 [scrapy] INFO: Optional features available: ssl, http11, boto
2018-12-04 11:05:45 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tumblrSpider.spiders', 'SPIDER_MODULES': ['tumblrSpider.spiders'], 'BOT_NAME': 'tumblrSpider'}
2018-12-04 11:05:45 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2018-12-04 11:05:45 [boto] DEBUG: Retrieving credentials from metadata server.
2018-12-04 11:05:45 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, LocalProxySpiderMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2018-12-04 11:05:45 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2018-12-04 11:05:45 [scrapy] INFO: Enabled item pipelines: TumblrspiderPipeline
2018-12-04 11:05:45 [scrapy] INFO: Spider opened
2018-12-04 11:05:45 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-04 11:05:45 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
myusername
2018-12-04 11:05:46 [scrapy] ERROR: Error downloading <GET https://myusername.tumblr.com/api/read/json?start=0&num=200>
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 555, in connect
timeout=self._timeout, bindAddress=self._bindAddress)
File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 482, in connectTCP
c = tcp.Connector(host, port, factory, timeout, bindAddress, self)
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 1165, in init
if abstract.isIPv6Address(host):
File "/usr/lib/python2.7/dist-packages/twisted/internet/abstract.py", line 522, in isIPv6Address
if '%' in addr:
TypeError: argument of type 'NoneType' is not iterable
2018-12-04 11:05:46 [scrapy] INFO: Closing spider (finished)
2018-12-04 11:05:46 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
'downloader/exception_type_count/exceptions.TypeError': 1,
'downloader/request_bytes': 513,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 12, 4, 11, 5, 46, 172847),
'log_count/DEBUG': 2,
'log_count/ERROR': 1,
'log_count/INFO': 7,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2018, 12, 4, 11, 5, 45, 960112)}
2018-12-04 11:05:46 [scrapy] INFO: Spider closed (finished)
项目有三个tumblrSpider目录,是在最深的那个目录执行么? 使用python2还是3,如果在国外vps运行,不需要代理的话要怎么配置,我只修改了初始地址和深度为2,跑下来出错,python2.
好久没用python了,看得头晕😵。谢谢