Hanging on file process about crawler4j HOT 4 CLOSED

ok, I tried your scenario but it works for me.

You are right that it skips both of those pages as they are bigger than 4mb but 
it does continue...

Please try crawling with logger configured to show DEBUG logs, do you see 
something additional ?


Here are my logs:
19:24:47 INFO  [main] - [CrawlController]- Crawler 1 started
19:24:47 INFO  [main] - [CrawlController]- Crawler 2 started
19:24:47 INFO  [main] - [CrawlController]- Crawler 3 started
19:24:49 INFO  [Crawler 1] - [WebCrawler]- URL: 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/?C=S%3BO%3DA
19:24:49 INFO  [Crawler 2] - [WebCrawler]- URL: 
http://www.ics.uci.edu/~yil8/public_data/
19:24:50 DEBUG [Crawler 2] - [WebCrawler]- Skipping: 
http://www.ics.uci.edu/icons/unknown.gif as it contains binary content which 
you configured not to crawl
19:24:50 WARN  [Crawler 2] - [WebCrawler]- Skipping a URL: 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/tumor.bam.bai which was bigger ( 
4523128 ) than max allowed size
19:24:50 INFO  [Crawler 3] - [WebCrawler]- URL: 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/
19:24:51 WARN  [Crawler 2] - [WebCrawler]- Skipping a URL: 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/normal.bam.bai which was bigger 
( 4534848 ) than max allowed size
19:24:51 INFO  [Crawler 3] - [WebCrawler]- URL: 
http://www.ics.uci.edu/~yil8/public_data/?C=D%3BO%3DA
19:24:51 INFO  [Crawler 3] - [WebCrawler]- URL: 
http://www.ics.uci.edu/~yil8/public_data/?C=N%3BO%3DD
19:24:51 INFO  [Crawler 2] - [WebCrawler]- URL: 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/?C=N%3BO%3DD
19:24:52 DEBUG [Crawler 2] - [WebCrawler]- Skipping: 
http://www.ics.uci.edu/icons/back.gif as it contains binary content which you 
configured not to crawl
19:24:52 INFO  [Crawler 3] - [WebCrawler]- URL: http://www.ics.uci.edu/~yil8/



And it continues on and on...

Ah, i was running 1 crawler instance, not 3.
Also i was using
@Override
public WebURL handleUrlBeforeProcess(WebURL curURL) {
    System.out.println("handling " +curURL.getURL());
      return curURL;
}

2015-01-28 23:56:14,063 INFO  [main] - 
[edu.uci.ics.crawler4j.crawler.CrawlController] - Crawler 1 started
2015-01-28 23:56:14,516 INFO  [Crawler 1] - 
[edu.uci.ics.crawler4j.crawler.WebCrawler] - 1 URL: 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/?C=S%3BO%3DA
2015-01-28 23:56:14,626 INFO  [Crawler 1] - 
[edu.uci.ics.crawler4j.crawler.WebCrawler] - 2 URL: 
http://www.ics.uci.edu/~yil8/public_data/
2015-01-28 23:56:14,896 WARN  [Crawler 1] - 
[edu.uci.ics.crawler4j.fetcher.PageFetcher] - Failed: Page Size (4523128) 
exceeded max-download-size (1048576), at URL: 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/tumor.bam.bai
2015-01-28 23:56:14,896 WARN  [Crawler 1] - 
[edu.uci.ics.crawler4j.crawler.WebCrawler] - Skipping a page which was bigger 
than max allowed size: 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/tumor.bam.bai
2015-01-28 23:56:15,302 WARN  [Crawler 1] - 
[edu.uci.ics.crawler4j.fetcher.PageFetcher] - Failed: Page Size (4534848) 
exceeded max-download-size (1048576), at URL: 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/normal.bam.bai
2015-01-28 23:56:15,302 WARN  [Crawler 1] - 
[edu.uci.ics.crawler4j.crawler.WebCrawler] - Skipping a page which was bigger 
than max allowed size: 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/normal.bam.bai

(obviously bad technique i didnt use the logger for preProcess.... but from 
what i recall last line was: "handling 
http://www.ics.uci.edu/~yil8/public_data/PyLOH/normal.bam" or tumor.bam
they are huge files, it was hanging for at least 45 minutes until i stopped it

(I circumvented .bam files for now, while i am crawling, but ill see if i can 
get a better log once my current crawl is done.)

I found the problem, It got caught on my trap avoidance algorithm.
It was pointing back to the same page with some links, with different urls.

Ok.


Thank you for the report though.