vidageek / crawler Goto Github PK

Decide what we should do when MimeType is null.

Currently, it raises a NPE.

the programe will be dead if running large scale task!!!

template
should match
and set test to be "a.gif"

DefaultNormalizer should concatenate relative url begin with '?' with the url which conatains it, not the base url.
For example, if the base url is http://www.some.com, when crawler follows the relative url "?pageno=3" in page http://www.some.com/sample, the DefaultNormalizer will return http://www.some.com?pageno=3,but not http://www.some.com/sample?pageno=3, which it should be.
I solved this problem by change the interface method signature from LinkNormalizer#String normalize( final String relativeUrl) to String normalize(final String urlToCrawl, final String relativeUrl), and in PageCrawlerExecutor#run() invoke normalizer.normalize(urlToCrawl.link(), l);

Hi, crawler.
I find may be there is one logic error in PageCrawlerExecutor's run() method.
" String link = normalizer.normalize(l); ", this will get link alawys relate baseUrl, but in pratice
we should get link every time relate last urlToCrawl.
I change it to " String link = Utils.normalize(urlToCrawl.link(), l); "
public static String normalize(final String beginUrl, String url) {
url = url.replaceAll("&", "&");
return UrlUtils.resolveUrl(beginUrl, url);
}

I do not whether it's error, but after change this code, i get the result i wanted.

More word, this project is good indeed. I like it.