Smart websites return messages like"... With JavaScript and cookies enabled... " about scrapegraph-ai HOT 12 CLOSED

I thought it was done but alas no,
https://www.coindesk.com/markets/2024/05/31/bitcoin-breaks-to-low-end-of-trading-range-but-june-data-could-be-next-catalyst/
returns
In a shocking turn of events, a website has been found with no body content, leaving readers puzzled. The mysterious disappearance of information has sparked curiosity and speculation. Stay tuned for updates on this developing story.

What gets me is the smugness of the site author!

another...
https://www.investing.com/news/cryptocurrency-news/colossal-253-billion-bitcoin-withdrawal-stuns-major-exchanges-3466201

from scrapegraph-ai.

but the link you provided is not a valid link, there is no this page.

from scrapegraph-ai.

but the link you provided is not a valid link, there is no this page.

missing a "d" in the link: https://cointelegraph.com/news/ethereum-due-new-all-time-high-ether-etf-nears-end

from scrapegraph-ai.

Ah thanks! I see the problem has been found minutes before I had the opportunity.
Sorry for the inconvenience.

I have just tried it again and got
{'summary': 'The story is about the need to enable JavaScript and cookies in order to continue on the website.'}

with prompt
prompt="Give a summary of the story",

from scrapegraph-ai.

Hey @Bandit253 try setting the headless flag to False in the graph_config, I got the right answer:

{'summary': "The article discusses the anticipation of Ethereum (ETH) reaching a new all-time high as the launch of spot Ether ETFs in the United States nears. Market analyst Michaël van de Poppe predicts that ETH/USD is likely to surpass its previous record peak, driven by the approval of these ETFs. This development is expected to reduce Bitcoin's market dominance, giving altcoins like Ethereum more room to grow. At the time of writing, ETH is trading at around $3,850, still below its record high of $4,900 set in late 2021. The article also mentions that BlackRock's IBIT has become the world's largest Bitcoin ETF, surpassing the Grayscale Bitcoin Trust (GBTC)."}

from scrapegraph-ai.

Fabulous, I do also...
thanks so much!!

from scrapegraph-ai.

try add a slow_mo param to the config like:

graph_config = {
    "llm": {
...
    },
    "loader_kwargs": {
        "slow_mo": 10000
    }
}

I found this in another issue.

from scrapegraph-ai.

Yes, some websites need time to get the code

from scrapegraph-ai.

I like and understand the thinking but alas the websites still elude. I got this from https://www.coindesk.com/markets/2024/05/31/bitcoin-breaks-to-low-end-of-trading-range-but-june-data-could-be-next-catalyst/

A gripping tale with no body content leaves readers on the edge of their seats. What secrets lie within?

I slowed it down by the suggested 10000 then 20000, with the same result.

graph_config = { "llm": {
                    "api_key": self.oAI_key,
                      "model": "gpt-3.5-turbo",
                        },
                            "loader_kwargs": {
                                "slow_mo": 20000
                            },
                            "headless": False,
                            }

As usual thanks for the tips!

from scrapegraph-ai.

try with 5k or 10k, just for trial

from scrapegraph-ai.

yep tried 50k and 100k same result. :(
Unrelated but interesting observation, I opened the same page in edge and while it is open it thrushes my C: drive!
The red arrow indicates when I opened the page and when I close the tab. Wondering if the BTC mining

from scrapegraph-ai.

hi, please install the new beta and try again the script

from scrapegraph-ai.