brendonboshell / supercrawler Goto Github PK

Hi great tool, is there any way to set an referrer like the url of the page that lead the crawler to the current page.

URL Destination
https://example.com https://example.com/page1

Also is there possible to make the crawler scan all links but store in the database only the external links ?

Thank you very much

I am a little confused over how the sitemap parser works.

var sp = new sitemapsParser({
      urlFilter: function (url) {
        return url.indexOf("de") === -1;
      }
    });

This looks like it finds all links that match a certain pattern in the sitemap itself (rather than a list of sitemaps) am I right?

I didn't find on documentation how can we handle when the crawler finished.

I'm considering extending supercrawler to support Phantom or https://github.com/GoogleChrome/puppeteer (i.e. Headless Chrome, as a modern alternative to PhantomJS) for the purposes of parsing dynamic content—do you have some ideas on how best to architect or re-architect this?

See #16 (comment): "Using the start method of Crawler after you have called stop is problematic because it doesn't properly handle the case where you resume while there are outstanding requests from the previous crawl."

pass custom headers, use cookie jar, etc.

I'm trying this: https://gist.github.com/t3db0t/2db07159130bc5c0dc9ebe6c8dc8f317

The only output I'm getting is:

$ node test.js 
Starting Crawler...
Crawling https://sweetpricing.com
Crawling https://sweetpricing.com/robots.txt
Crawling https://sweetpricing.com/en/

And then it hangs indefinitely. Any ideas? Is there any debug output option?

hi Brendon!

1. thanks for making this!

2. what is the best way to make a Crawl that is in-progress, PAUSE?
   ...and then (when the user decides ) to Continue from exactly where it left off?

cheers!!

I might have missed it, but in the crawler options i didnt see a way to control how deep the crawler goes. Is that possible via a parameter, or would i have to implement that logic in the htmllinkParser handler via urlFilter?

I'd like to see when one of supercrawler's handlers queue up another URL to crawl. Would be nice to add an event for this.

Basically we need to get all internal links of our user's websites so we only want to crawls internal links within supplied url.

Is it possible to configure supercrawler like that?

[email protected] introduces support for Node.js 10.

Problem: Script doesn't exit when using RedisUrList.

Step to replicate:
Run the following code:

'use strict';

const supercrawler = require('supercrawler');
const crawler = new supercrawler.Crawler({
    urlList: new supercrawler.RedisUrlList({
        redis: {
            host: 'redis-server.example.org'
        }
    })
});

console.log('Script should exit after this.');

Expected behavior:
Script should stop after running.

Actual behavior:
Script runs indefinitely.

Workaround:
Call process.exit() to terminate the script.

BTW, I'm using AWS ElastiCache for Redis, just in case this detail is needed. :)

I'm currently using supercrawler for typescript. I described the base types in my repository. Could you add them to the main application?

I think that you need to slightly work them out before adding. I hope for your help

Hello, thanks for the app, I was using it to crawl links of my website but I found that it add statusCode NULL to crawled links in the database while there is no errors at all.

Why does it act like that, I consider a request is successful and should log it's success statusCode if there were no errors, also strangely not all of succeeded requests gives statusCode of NULL.

I have checked the access log also during the crawler work and found 200 responses to URLs that logs statusCode of NULL

Hi, some websites cannot be crawled because are blocking the access, how can I implant a proxy to be used with the crawler?

Thank you

Sequelize at the version currently specified has an SQL injection vulnerability. This should be updated ASAP to a version >= 5.8.11.

It doesn't seem to work, and I can't find anywhere in the source that actually utilizes it (just checks to see if it's there?)

supercrawler is picking up ALL links on a page. If there are links to movie files, images, or any large files it will add these URLs to the queue. The urls get passed to request which tries to download them.

The url is https://www.fomento.gob.es/informacion-para-el-ciudadano/participacion-publica/sec-carreter

Only recover the first html but not ther links. Why?

I try to crawl a website but there is a captcha appearing after a certain amount of page crawled. So i open my browser and fill the captcha manually.
Is there a way to avoid to do that or at least increase the number of page i can crawler before get this message. cooldown? hwo to?

I see in the database there is a field for nextRetryDate which i assume is used for failed requests.

How would one make it so that a domain/url can be "re-crawled" let's say every day?

At the moment, once a url is finished, it won't touch it again.

In addition to limit by domain name is it possible to limit to a specific path and sub path url
For the moment im doing this into an handler but it register urls into urls table even if I dont use it.

First of all thanks a lot for this nifty crawler. I am really enjoying your api design!

Nevertheless, would it be possible to expose the entire response object in the handler api? Currently supercrawler only exposes contentType, response.body and url.

This would allow us to crawl a lot more information:

• crawl response headers
• handle http status codes/messages
• performing custom redirects

I can create a PR if such a change would be acceptable.

Hi, how can I crawl only last x URL's found on the homepage ?

When specifing a hostname to restict to like "www.acme.com", and a path like: "www.acme.com/foo" return a 301 the location is added to the queue without validation that it has the correct hostname.

Maybe a "hook" should be implemented here: https://github.com/brendonboshell/supercrawler/blob/master/lib/Crawler.js#L192

Allowing the htmlLinkParser to intercept and ignore the upsert.