Bypass protection about advertools HOT 3 CLOSED

In fact I am making a test with zite.
cf https://docs.zyte.com/zyte-api/get-started/tutorial/index.html#start-a-scrapy-project

how can I set in custom_settings the following infos :
DOWNLOAD_HANDLERS = {
"http": "scrapy_zyte_api.ScrapyZyteAPIDownloadHandler",
"https": "scrapy_zyte_api.ScrapyZyteAPIDownloadHandler",
}
DOWNLOADER_MIDDLEWARES = {
"scrapy_zyte_api.ScrapyZyteAPIDownloaderMiddleware": 1000,
}
REQUEST_FINGERPRINTER_CLASS = "scrapy_zyte_api.ScrapyZyteAPIRequestFingerprinter"
ZYTE_API_KEY = "YOUR_API_KEY"
ZYTE_API_TRANSPARENT_MODE = True

from advertools.

Any hint to bypass protection like cloudflare?

Cloudflare is usually tough, but there are a few ways you can do that:

• Slow down your crawling
• Change your User-agent
• Use proxies

Details on the above and a bunch of crawl strategies can be found here:

https://advertools.readthedocs.io/en/master/advertools.code_recipes.spider_strategies.html

how can I set in custom_settings the following infos :

If you are using advertools, you can pass them all to the custom_settings parameter:

adv.crawl(
    'https://example.com', 'output.jl',
    custom_settings={
    'DOWNLOAD_HANDLERS': {
        "http": "scrapy_zyte_api.ScrapyZyteAPIDownloadHandler",
        "https": "scrapy_zyte_api.ScrapyZyteAPIDownloadHandler",
},
'DOWNLOADER_MIDDLEWARES': {
"scrapy_zyte_api.ScrapyZyteAPIDownloaderMiddleware": 1000,
},
'REQUEST_FINGERPRINTER_CLASS': "scrapy_zyte_api.ScrapyZyteAPIRequestFingerprinter",
'ZYTE_API_KEY': "YOUR_API_KEY",
'ZYTE_API_TRANSPARENT_MODE': True
})

If you are using scrapy these would be provided as command line or other available options.

Hope this helps.

from advertools.

Great ! I'll test

from advertools.