genre_scraper.py only scraping 4000 images max about art-dcgan HOT 9 OPEN

I'll look into this more over the weekend - really sorry it doesn't work, and thanks for bringing it to my attention - leaving this thread open until i fix it...

from art-dcgan.

try it now - i just updated the scraper

from art-dcgan.

I've tried everything and couldn't fix this. :(
You updated the code but only changed one import, I don't think it does anything. The sites only shows 3600 pictures per style/genre.

from art-dcgan.

Thanks for the quick update but it still only attempts to load 3915 pictures, tried with different num_pages values but no avail.

from art-dcgan.

I am having the same problem. If this is not resolvable, would it be possible for you to upload the complete set of images that I assume you still have stored somewhere to a google drive folder? It would be incredibly appreciated. Cheers

from art-dcgan.

@JOHN-MARSH i'm still looking into it - i think it might be a question of too many threads working at once... i think it will be resolvable.

from art-dcgan.

Cheers mate. I would try to fix it myself, but web scraping is not something that I have experience with. Keep us updated!

from art-dcgan.

any updates?

from art-dcgan.

Not sure if this is a related issue, but I had problem scraping image names that are not utf-8 compatible because it had accent characters. I fixed the problem by adding urllib.parse.quote under def downloader as follows,

file=urllib.parse.quote(file, safe=':/')
filepath = file.split('/')

from art-dcgan.