Code Monkey home page Code Monkey logo

cnkispider's Introduction

CNKISpider

知网专利爬虫,仅用于学习交流,不做商业使用

发现新的爬取入口

今天同学突然告诉我爬取了100多W(我们需要爬2014年的,总共190W+),细问才知道,知网的专利详情页的url组成是有规则的。

举个例子:

http://dbpub.cnki.net/grid2008/dbpub/Detail.aspx?DBName=SCPD2014&FileName=CN203968251U&QueryID=28&CurRec=2

对于这个某个专利的url来说,我们只要变化FileName=CN203968251U就可以了,=号后面代表的是专利公开号,专利公开号亦称专利文献号,组成方式为“国别号+分类号+流水号+标识代码”,如CN1340998A,表示**的第340998号发明专利(来自百度百科)。

假如我们需要爬取2014年的所有专利,我们可以通过搜索找到2014年1月1日(2014年非常早的一篇专利号)和2014年12月31日(2014年非常晚的一篇专利号),取中间的差值,就可以爬取绝大部分需要的专利了。

其中,CN是固定的,末尾的字母是专利标识代码,**只有ASU种

所有,避免了爬取url列表页(反爬虫严重)和复杂的验证码问题,直接构建循环爬取详情页即可。

项目使用工具

框架使用的的Scrapy1.3,python版本3.6

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.