thatlittlegit-archive / webcrawler Goto Github PK

get_urls_in_url is monolithic - our entire web-crawler is basically this function in a loop. We need to separate it, preferably in different files, so we can implement more. For example, we need to log the data the crawler collects; but we would have to shove it somewhere in the indented pigsty that is called 'get_urls_in_url'. We also need to rename it - it sounds like it finds https://google.com in https://bing.com/https://google.com/.

get_root_domain ought to be refactored. Such a simple function, which I'm sure that could be compressed, is around 38 lines! We need to implement more Rust instead of basically porting JavaScript. It might help speed, too.

Crawler should be able to find URLs in txt files. Right now, they're indexed, but not parsed. This should be fixed, but I probably won't add it in the near future.

Crawler should be able to find URLs in CSS files (i.e. url(...)). Right now, they're indexed, but not parsed. This should be fixed, but I probably won't add it in the near future.

html5ever is built by the Servo project, and thus is probably faster. Maybe we should try switching? I think we should do a speed comparison - I doubt the differences will be large.

I've attempted to write a test for _main_loop, using Iron. However, it seems to hang while creating a reqwest::Client. The code is currently commented out, but it needs to be fixed.

This is currently waiting on this Stack Overflow question.

Crawler should be able to find URLs in JavaScript files (specifically strings). Right now, they're indexed, but not parsed. This should be fixed, but I probably won't add it in the near future.