Drain queue about fetchbot HOT 8 CLOSED

q.Cancel sounds like a good idea, yeah. I may not get to it before a few weeks, though.

from fetchbot.

Actually I had a little free time and I implemented an experiment on the branch cancel if you want to take a look. I may have missed a couple things, haven't looked at that code in a while, but if that's it it would be pretty simple. I will test it when I have a bit more time.

from fetchbot.

Did you get a chance to try it out? Was it working as expected?

from fetchbot.

I got a chance to try it out, but not to debug it (got on a plane and am now in a different city doing different things).

It appears to work (it cancels further crawling), but never actually unblocks.

Stops here:

My queue was fairly long at time of cancellation (several thousand URLs at a default rate of 2 reqs per second), but it hangs indefinitely on waiting for something else to mark the waitgroup as done.

from fetchbot.

Cool, I'll try to look at it today.
On Thu, Jul 23, 2015 at 12:18 David Kitchen [email protected]
wrote:

I got a chance to try it out, but not to debug it (got on a plane and am
now in a different city doing different things).

It appears to work (it cancels further crawling), but never actually
unblocks.

Stops here:

fetchbot/fetch.go

Line 142 in 96b7aa9

q.wg.Wait()

My queue was fairly long at time of cancellation (several thousand URLs at
a default rate of 2 reqs per second), but it hangs indefinitely on waiting
for something else to mark the waitgroup as done.

—
Reply to this email directly or view it on GitHub
#10 (comment)
.

from fetchbot.

Much appreciated, in fact if you have a PayPal email let me know. Beer will be bought.

from fetchbot.

Thanks, I appreciate it, but I'm lucky enough to not be needing donations so feel free to give that amount to a charity of your choice. In case you have no preference, those are important to me:

https://netdonor.net/ea-action/action?ea.client.id=1770&ea.campaign.id=17989

https://greenpeace.donorportal.ca/Donation/DonationDetails.aspx?L=en-CA&G=131&F=985&T=RECUR&cscid=Homepage_Donate_Button_Eng&__utma=218051913.31581654.1437687123.1437687123.1437687123.1&__utmb=218051913.4.10.1437687123&__utmc=218051913&__utmx=-&__utmz=218051913.1437687123.1.1.utmcsr=(direct)%7Cutmccn=(direct)%7Cutmcmd=(none)&__utmv=-&__utmk=147148404

from fetchbot.

I found the bug and added a test for the Cancel method, it should now be working as expected and finishing quickly. Let me know how it goes and I'll merge once you confirm it's good for you too (if you're able to test it, of course, otherwise I'll just merge right away).

from fetchbot.