`HcfMiddleware` consumes too many memory on large number of URLs about scrapylib HOT 5 OPEN

I'll be working on a PR for solving this issue.

from scrapylib.

I don't think your memory issues come from this middleware.

1. Are you using SitemapSpider along with this middleware?
2. Are you reading requests as you upload them with the same spider?

from scrapylib.

Sorry I mistakenly clicked on the "Close and comment" button just now.. :(

Thanks for the reply @nramirezuy :)

1. Yes. (But hacked into its _parse_sitemap and _parse_urlset methods)
2. No.
   I'm working with a hacked SitemapSpider, a typical job sends ~500 requests and got ~10 million URLs to store to HCF. Wouldn't there be ~10 million URLs stored into HcfMiddleware.new_links? What I want is to eliminate this overhead. :)

from scrapylib.

What do you mean by hacked, what are the changes made to the spider? SitemapSpider is extremely memory consuming because of big sitemap files, which implies massive responses.

I would suggest to not remove new_links just to change it to use the same technique Scheduler uses and make it optional by a setting.

from scrapylib.

What do you mean by hacked, what are the changes made to the spider? SitemapSpider is extremely memory consuming because of big sitemap files, which implies massive responses.

Setting request.meta["use_hcf"] = True for requests that come from the sitemap. Thus there won't be massive responses, only massive URLs to be stored to HCF.

I would suggest to not remove new_links just to change it to use the same technique Scheduler uses and make it optional by a setting.

Sure. This is exactly what I do in PR #57 :)

from scrapylib.