starfalll / spider Goto Github PK
View Code? Open in Web Editor NEW新浪微博爬虫(Sina weibo spider)，百度搜索结果 爬虫
License: MIT License
spider's People
Contributors
Stargazers
Watchers
Forkers
icewiki awesome-archive titanrisen balloontmz wangjianyuweg leexiaosong908 a794581572 andy2000xyz heartbreaker97 li123 qixibi m4cr0-storm walkerwell astudyingboy ltlullaby jerryfan86 songjianbin lovesnacks landeqi hhy5277 danyueming keanchen2019 citywill mrmiaolei nicky-chu wangpanqiao bangelzhb cell13 zhangjingfeng520 xdongsheng jeaken huoshi5151 zhangmq13thu cha0res scpoa ccviolet juuulia17 jiaxin-pei suinainksm wyj0613 luzy99 jr--chen eternity666666 zxhycxq sub1s y01zheng ganjiu11 hongliangbest shinobucr z-maple tuoxin126 octopus816 bianchengge yunshu2009 wulin-challenge 1baijiespider's Issues
运行weibo_spider.py报错
会报一个这样的错：FileNotFoundError: [Errno 2] No such file or directory: 'cookies.pkl'
报错具体如下：
(WeiboSpider)jiang@ubuntu:~/Spider/weibo$ sudo python3.5 weibo_spider.py Traceback (most recent call last): File "weibo_spider.py", line 205, in <module> main() File "weibo_spider.py", line 194, in main cookies = pickle.load(open('cookies.pkl', 'rb')) FileNotFoundError: [Errno 2] No such file or directory: 'cookies.pkl'
你好 咨询关于爬微博和qq空间的问题
付费咨询关于新浪微博和qq空间的问题 留一个微信
请赐教
您好
我在实现的过程中遇到了问题，想请您指点一下，信息，方便的话请加我的微信：yandayileng
谢谢。
大佬，百度请求页面只能获取第一页，后面的页面需要验证
我的搜索如下：
data = {'wd': "intitle:疫情 intitle:武汉 site:www.caixin.com", 'tn': 'baiduhome_pg', 'ie': 'utf-8', 'bsst': 1, 'pn': str(i - 1) + '0', }
第一页返回非常正常，获取到的链接，比如下一页的链接，再用page=requests.get(next_page_url, headers=headers)
就无法获取有效信息了，soup的值打印如下：
应该是一个验证码的问题。贼难受，如果大佬之前遇到过，恳请大佬指导！
`soup: <!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="utf-8"/>
<title>ç¾åº¦å®å
¨éªè¯</title>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
<meta content="yes" name="apple-mobile-web-app-capable"/>
<meta content="black" name="apple-mobile-web-app-status-bar-style"/>
<meta content="width=device-width, user-scalable=no, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0" name="viewport"/>
<meta content="telephone=no, email=no" name="format-detection"/>
<link href="https://www.baidu.com/favicon.ico" rel="shortcut icon" type="image/x-icon"/>
<link href="https://www.baidu.com/img/baidu.svg" mask="" rel="icon" sizes="any"/>
<meta content="IE=Edge" http-equiv="X-UA-Compatible"/>
<meta content="upgrade-insecure-requests" http-equiv="Content-Security-Policy"/>
<link href="https://wappass.bdimg.com/static/touch/css/api/mkdjump_8befa48.css" rel="stylesheet">
</link></head>
<body>
<div class="timeout hide">
<div class="timeout-img"></div>
<div class="timeout-title">ç½ç»ä¸ç»å¼è¯·ç¨åéè¯</div>
<button class="timeout-button" type="button">è¿å¦é¡µ</button>
</div>
<div class="timeout-feedback hide">
<div class="timeout-feedback-icon"></div>
<p class="timeout-feedback-title">é®é¢é¦</p>
</div>
<script src="https://wappass.baidu.com/static/machine/js/api/mkd.js"></script>
<script src="https://wappass.bdimg.com/static/touch/js/mkdjump_2e06726.js"></script>
</body>
</html>`
大神，我用您的代码学习，运行的时候insert总是报错， 这是什么原因勒，AttributeError: 'Insert' object has no attribute 'on_duplicate_key_update' 我用的mysql版本是8 python3.6
cookies.pkl
这个文件是干什么的，是生成的还是需要自己去创建的
AttributeError: 'str' object has no attribute 'get'
大佬，在运行Data-analysis的时候会报错
Traceback (most recent call last):
File "E:/python/python项目/Spider-master/weibo/data_analysis/Data_analysis.py", line 150, in
main(uid)#指定需要分析的用户的uid（必须先存在conf.yaml里面，并且运行了一次sina_spider程序），默认为conf.yaml中的第一条uid
File "E:/python/python项目/Spider-master/weibo/data_analysis/Data_analysis.py", line 135, in main
plot_create_time(time_lists)
File "E:/python/python项目/Spider-master/weibo/data_analysis/Data_analysis.py", line 128, in plot_create_time
chart = Bar('用户微博动态发布时间')
File "E:\python\lib\site-packages\pyecharts\charts\chart.py", line 163, in init
super().init(init_opts=init_opts)
File "E:\python\lib\site-packages\pyecharts\charts\chart.py", line 14, in init
super().init(init_opts=init_opts)
File "E:\python\lib\site-packages\pyecharts\charts\base.py", line 29, in init
self.width = _opts.get("width", "900px")
AttributeError: 'str' object has no attribute 'get'
在百度上找的原因好像都不对，请问怎么解决啊。
Data_analysis.py运行问题
大佬好，请问一下我在将用户的数据爬取下来保存到Mysql以后，怎么运行Data_analysis.py文件会出现
Traceback (most recent call last):
File "E:/python/python项目/Spider-master/weibo/data_analysis/Data_analysis.py", line 150, in
main(uid)#指定需要分析的用户的uid（必须先存在conf.yaml里面，并且运行了一次sina_spider程序），默认为conf.yaml中的第一条uid
File "E:/python/python项目/Spider-master/weibo/data_analysis/Data_analysis.py", line 135, in main
plot_create_time(time_lists)
File "E:/python/python项目/Spider-master/weibo/data_analysis/Data_analysis.py", line 117, in plot_create_time
long_lists.append(res[0])
IndexError: list index out of range
尝试打印了一下res，发现确实是空的，请问一下怎么解决
您好，请教新浪微博爬虫的问题
大佬您好，请问我运行您新浪微博爬虫的Create_all.py文件，显示ModuleNotFoundError: No module named 'sqlalchemy'，该怎么解决呢？谢谢！
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.