Code Monkey home page Code Monkey logo

reptiles's Introduction

爬虫项目

免责声明

所有项目仅用于学术研究，但因在**大陆频频出现爬虫开发者涉诉与违规相关的新闻

使用者需遵守其所在地的相关法律法规。因违法违规使用造成的一切后果，使用者自行承担

爬虫集合

  • 17K小说网
  • BOSS直聘
  • B站用户信息
  • 斗鱼直播数据
  • 豆瓣读书信息
  • 京东商品数据
  • 酷我音乐
  • 上证快讯
  • 网易云
  • 淘宝商品
  • 王者荣耀皮肤图片

主要技术

  • 使用requests库，来进行对网页的访问
  • 使用lxml库来对HTML进行解析使用xpath来提取内容信息
  • 使用pymysql库对数据库进行连接插入数据
  • 使用xlwt库进行创建工作簿存储数据

网易云

使用的selenium自动化，因为网易云是动态通过xpath无法获取， xpath只针对静态页面
selenium同样也提供了find_elements_by_xpath的方法来获取到对应的元素

上证快讯

使用了mysql来存储数据，每分钟访问一次并提取最新的信息来写入数据库，这样方便提取，使用Excel存储不能进行
实时提取，存入MySQL可以一边查看爬取到的信息另一边又不会影响到程序的正常运行

reptiles's People

Contributors

xiaoyanhui avatar

Watchers

avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.