关于账号全部被BAN后的继续执行任务问题 about weibospider HOT 9 CLOSED

root 1311 0.1 2.8 188344 57460 pts/8 S 13:04 0:00 /usr/bin/python3 /usr/local/bin/celery beat -A tasks.workers -l info
root 1379 0.0 0.0 14220 1084 pts/8 S+ 13:12 0:00 grep --color=auto celery
[1]- Exit 1 nohup celery -A tasks.workers -Q login_queue,user_crawler,fans_followers,search_crawler,home_crawler worker -l info -c 1

这条语句被exit是为什么？

from weibospider.

root@iZ2zeftexcphcu8dj9if0mZ:/home/admin/project/weibospider# jobs -l
[2]- 1311 Running nohup celery beat -A tasks.workers -l info &
[3]+ 1398 Running nohup celery -A tasks.workers -Q login_queue,user_crawler,fans_followers,search_crawler,home_crawler worker -l info -c 1 &

这个是正常的时候，不正常爬取的时候，jobs中就只剩下 nohup celery -A tasks.workers -Q 了
这是因为什么呢

from weibospider.

1.你检查一下是否能用你的账号进行搜索，因为微博封账号的情况很复杂，它可能只封锁你一个功能

2.如果你重启celery worker之后，它会直接继续执行上次没执行完的任务，你如果要让它从当前时刻开始执行你指定的任务，需要清除redis db5和6

from weibospider.

我试了试没有被封，之前enable从1变0是不是封了我几个小时？我现在update账号enable了以后，爬取时候，这条语句老是被exit，任务就停止了：

celery -A tasks.workers -Q login_queue,user_crawler,fans_followers,search_crawler,home_crawler worker -l info -c 1

from weibospider.

1.你在basic.py中打印一下response.status_code和response.text，看看响应是否是正常的

2.把redis的db1(cookies)、db5和6都清空，再启动worker和相关任务调度器

from weibospider.

2018-03-28 14:11:50 - crawler - INFO - the crawling url is http://weibo.com/p/1005052244164900/follow?page=1#Pl_Official_HisRelation__60
[2018-03-28 14:11:50,662: INFO/ForkPoolWorker-1] the crawling url is http://weibo.com/p/1005052244164900/follow?page=1#Pl_Official_HisRelation__60
2018-03-28 14:11:50 - crawler - ERROR - failed to crawl http://weibo.com/p/1005052244164900/follow?page=1#Pl_Official_HisRelation__60，here are details:MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error., stack is File "/home/admin/project/weibospider/decorators/decorator.py", line 14, in time_limit
return func(*args, **kargs)

[2018-03-28 14:11:50,664: ERROR/ForkPoolWorker-1] failed to crawl http://weibo.com/p/1005052244164900/follow?page=1#Pl_Official_HisRelation__60，here are details:MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error., stack is File "/home/admin/project/weibospider/decorators/decorator.py", line 14, in time_limit
return func(*args, **kargs)

[2018-03-28 14:11:50,667: WARNING/ForkPoolWorker-1] /root/anaconda3/lib/python3.6/site-packages/pymysql/cursors.py:166: Warning: (1062, "Duplicate entry '6490414635' for key 'uid'")
result = self._query(query)
[2018-03-28 14:11:50,667: WARNING/ForkPoolWorker-1] /root/anaconda3/lib/python3.6/site-packages/pymysql/cursors.py:166: Warning: (1062, "Duplicate entry '3764351355' for key 'uid'")
result = self._query(query)
2018-03-28 14:11:50 - crawler - INFO - the crawling url is http://weibo.com/p/1005053105868177/info?mod=pedit_more
[2018-03-28 14:11:50,682: INFO/ForkPoolWorker-1] the crawling url is http://weibo.com/p/1005053105868177/info?mod=pedit_more
2018-03-28 14:11:50 - crawler - ERROR - failed to crawl http://weibo.com/p/1005053105868177/info?mod=pedit_more，here are details:MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error., stack is File "/home/admin/project/weibospider/decorators/decorator.py", line 14, in time_limit
return func(*args, **kargs)

[2018-03-28 14:11:50,683: ERROR/ForkPoolWorker-1] failed to crawl http://weibo.com/p/1005053105868177/info?mod=pedit_more，here are details:MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error., stack is File "/home/admin/project/weibospider/decorators/decorator.py", line 14, in time_limit
return func(*args, **kargs)

[2018-03-28 14:11:50,684: ERROR/ForkPoolWorker-1] list index out of range
[2018-03-28 14:11:50,684: ERROR/ForkPoolWorker-1] list index out of range
[2018-03-28 14:11:50,685: ERROR/ForkPoolWorker-1] list index out of range
[2018-03-28 14:11:51,745: WARNING/MainProcess] Restoring 4 unacknowledged message(s)

原因好像是因为redis，但是我不熟悉redis所以不知道怎么改，这个是日志

from weibospider.

我查了一下，好像原因是因为强制把redis快照关闭了导致不能持久化的问题，在网上查了一些相关解决方案，通过stop-writes-on-bgsave-error值设置为no即可避免这种问题。

后续继续测试下还会不会停

from weibospider.

好的，多谢反馈

from weibospider.

MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
这个问题也有可能是内存占用率太高导致的, 我的VPS上是这个原因

from weibospider.