Questions about custom spider about advertools HOT 5 CLOSED

Is there a way to just extract the information I want, by default it extracts too much, if the web page is large, the json line file will be very large.

Yes, I'm aware it's a lot of information, and that's to make it as complete as possible for SEO/business/content analysis.
The solution is very simple, after crawling, you can read the output file and get the columns you want. Then delete the larger file.

Do you plan to add feature to create sitemap for this app. I have a lot of big websites that need to create a sitemap

One approach might be to use the crawler, which gives you all the URLs, and from that you can create a sitemap with a simple loop.
Do you have another idea of how that might work?

Hope that helps.
Thanks

from advertools.

Maybe I have to customize the code
I am currently creating sitemap as you said, I use 1 python file to create sitemap manually.
I am currently using screamingfrog for web scraping, I think advertools can replace it, just need to add some more features

from advertools.

I am currently creating sitemap as you said, I use 1 python file to create sitemap manually.

Great.

just need to add some more features

Cool. What do you suggest?

from advertools.

Can you guide me to configure the proxy?
I am currently using proxychains

from advertools.

You can check out some of Scrapy's content:

https://www.zyte.com/blog/scrapy-proxy/

https://docs.scrapy.org/en/latest/_modules/scrapy/downloadermiddlewares/httpproxy.html

https://zytedata.medium.com/how-to-set-up-a-custom-proxy-in-scrapy-d245a49784fd

from advertools.