darthbear / scrapy-proxynova Goto Github PK

We could track last 10 response times in deque for each proxy. And from time to time resort them according to the average response time.

Selection should use gaussian distrubution, like that:

from random import gauss

proxies = [some list where fastest proxies go first]

index = int(round(gauss(0, len(proxies) / 3.0)))
index = min(index, len(proxies) - 1)

return proxies[index]

http://www.proxynova.com/proxy_list.txt?country=us

Not working response 404 not found

I got this error while running the example project
Error: Cannot find any available proxies
I tried accessing the proxy page from my browser but proxy nova said that this list is currently offline so i think it's proxynova issue rather than scrapy-proxynova.

If scrapy is run multiple times in separate processes, we want to prevent them to all do the same job and try to generate a proxy list. Rather the first one should generate the list and the other ones wait for it to complete before starting crawling.