Bypass a cookie wall about advertools HOT 4 CLOSED

Hello Elias,
I did not know playright before. Works fine for bypassing js wall ! Thanks a million

from advertools.

Thanks @caroheymes

Indeed I haven't seen your previous issue. Not sure why. Sorry!

As a quick check, I just tried to crawl with advertools, and it seems OK (I tested 100 pages)

Here is the sample file:

https://docs.google.com/spreadsheets/d/12E-Sq3mELZPBS3RwZuyev3t5ClHBCQiAqIzNgvuPm4I

I didn't dig deep into the page, but it seems there is some JS causing the popup, and then it's blocking the page. But on the live website it works properly as in the shared file.

Just curious, why do you want to view it offline?

from advertools.

Hello Elias, thanks for your feedback.
I am building a streamlit app to benchmark competitors prices.

Everyday, agents visit & collect the prices of different players and push the data in bigquery.

I need also to identify which products are unavailable. I've tried several xpath cf example below and defintely, I thing that I am blocked by the cookie wall (cf view response in scrapy)

urls = 'https://www.interflora.fr/p/roses-passion' output_file_details = 'export.jl' adv.crawl(url_list = urls, output_file=output_file_details, xpath_selectors={'button' : '//span[@data-test-id = "button-add-to-cart"].text()'}, custom_settings={'ROBOTSTXT_OBEY': False, 'DOWNLOAD_DELAY' : 1, 'USER_AGENT':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'} df = pd.read_json(output_file_details, lines=True) df.columns => Float64Index([], dtype='float64')
the ld+json is unfortunately not enough

Thanks for your feedback

from advertools.

I think your selector is correct. But it seems this button is generated through some JS because its value is dynamic. I can't find it in the page source (but it's there when you inspect the element):

The word "épuisé" is itself épuisé :)

Checkout playwright maybe or other solutions. I'm thinking of options to tackle such situations.
Hope this makes sense.

from advertools.