axetroy / crawler Goto Github PK

View Code? Open in Web Editor NEW

41.0 1.0 9.0 629 KB

nodejs 爬虫框架. crawler framework for nodejs

Home Page: http://axetroy.github.io/crawler

License: Other

TypeScript 100.00%

nodejs crawler

crawler's Introduction

Nodejs 的爬虫框架

一个快速，简单，易用的 node.js 爬虫框架.

WARNING: 当前仍处于开发阶段，任何存在的特性都可能会更改或者移除

快速开始

npm install @axetroy/crawler

import { Crawler, Provider, Response } from "@axetroy/crawler";

class MyProvider implements Provider {
  name = "scrapinghub";
  urls = ["https://blog.scrapinghub.com"];
  async parse($: Response) {
    const $nextPage = $("a.next-posts-link").eq(0);

    if ($nextPage) {
      $.follow($nextPage.prop("href"));
    }

    return $(".post-header>h2")
      .map((_, el) => $(el).text())
      .get();
  }
}

const config: Options = {
  timeout: 1000 * 5,
  retry: 3
};

new Crawler(MyProvider, config)
  .on("data", (articles: string[]) => {
    for (const article of articles) {
      process.stdout.write(article + "\n");
    }
  })
  .on("error", (err, task) => {
    console.log(`request fail on ${task.url}: ${err.message}`);
  })
  .start();

crawler's People

Contributors

Stargazers

Watchers

Forkers

youmengme torns yifan-blog sangecoder ideas-design yinjunwen framework-design dec-qxy alex1504

crawler's Issues

案例运行错误，RangeError: Use `yyyy` instead of `YYYY`（date-fns）

import { Crawler, Provider, Response, Options } from "@axetroy/crawler";

class MyProvider implements Provider {
  name = "scrapinghub";
  urls = ["https://blog.scrapinghub.com"];
  async parse($: Response) {
    const $nextPage = $("a.next-posts-link").eq(0);

    if ($nextPage) {
      $.follow($nextPage.prop("href"));
    }

    return $(".post-header>h2")
      .map((_, el) => $(el).text())
      .get();
  }
}

const config: Options = {
  timeout: 1000 * 5,
  retry: 3,
};

new Crawler(MyProvider, config)
  .on("data", (articles: string[]) => {
    for (const article of articles) {
      process.stdout.write(article + "\n");
    }
  })
  .on("error", (err, task) => {
    console.log(`request fail on ${task.url}: ${err.message}`);
  })
  .start();

error

(node:47382) UnhandledPromiseRejectionWarning: RangeError: Use `yyyy` instead of `YYYY` for formatting years; see: https://git.io/fxCyr
    at throwProtectedError (/Users/guanwei/code/demo/spider/wb_xhs_media/node_modules/date-fns/_lib/protectedTokens/index.js:22:11)
    at /Users/guanwei/code/demo/spider/wb_xhs_media/node_modules/date-fns/format/index.js:429:41
    at Array.map (<anonymous>)
    at Object.format (/Users/guanwei/code/demo/spider/wb_xhs_media/node_modules/date-fns/format/index.js:413:45)
    at Printf.template (/Users/guanwei/code/demo/spider/wb_xhs_media/node_modules/@axetroy/crawler/src/logger.ts:16:18)
    at Printf.transform (/Users/guanwei/code/demo/spider/wb_xhs_media/node_modules/logform/printf.js:11:26)
    at Format.transform (/Users/guanwei/code/demo/spider/wb_xhs_media/node_modules/logform/combine.js:20:24)
    at DerivedLogger._transform (/Users/guanwei/code/demo/spider/wb_xhs_media/node_modules/winston/lib/winston/logger.js:305:29)
    at DerivedLogger.Transform._read (/Users/guanwei/code/demo/spider/wb_xhs_media/node_modules/readable-stream/lib/_stream_transform.js:177:10)
    at DerivedLogger.Transform._write (/Users/guanwei/code/demo/spider/wb_xhs_media/node_modules/readable-stream/lib/_stream_transform.js:164:83)
(node:47382) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:47382) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Will it be to request a feature to bypass cloudflare recaptcha? This might help https://github.com/codemanki/cloudscraper .
I have my crawler built with https://github.com/matthewmueller/x-ray and cloudscraper and it works for me.
But I'll be doing a big rewrite of the system so I'm looking for alternatives and this project has caught my eye.