Comments (14)
I can confirm the issue. Cloudflare is taking a very long time to resolve. I tested and could observe the following behavior:
- it eventually resolves, but it can take up to multiple minutes.
- the cloudflare page reloads multiple times. Increasing the overall timeout for all methods works, but not sure how usefull the scraping is when it takes ~2 mins to resolve.
Deploying a quickfix soon that works with extended execution time of up to 2 mins (not optimal though). If anyone has ideas how to resolve cloudflare please share.
I found this repo, will test and report soon:
https://github.com/JimmyLaurent/cloudflare-scraper
from opensea-scraper.
Same here.. try looking for solutions
from opensea-scraper.
ok cloudflare-scraper repo I tested does not work, it broke since cloudflare started to reload the page (which we also experience):
JimmyLaurent/cloudflare-scraper#39 (comment)
Unfortunately the waiting for 2 mins somehow also does not work consistently. Cant figure out why for now. Not sure what else to do.
I have to say sometimes cloudflare is getting more rigid and its a temporary thing, so chances are opensea experiences a ton of traffic and thats why cloudflare is more rigid, but we'll have to see if this issue persist long term.
Sorry theres no fix for now :( if anyone has ideas please share!
from opensea-scraper.
would disabling timeout on waitForSelector work?
the error im getting is
waiting for selector
.cf-browser-verification to be hidden failed: timeout 30000ms exceeded
- guess they extended at a variable time..
also perhaps should wait for #__next tag to show?
from opensea-scraper.
waitForSelector
has a default timeout for 30000ms (30 secs). I tried extending to 2 mins but for me it still did not work. I also wanted to wait for the opensea page to appear, but with no luck. You can try around yourself:
// REPLACE
await page.waitForSelector('.cf-browser-verification', {hidden: true});
// WITH
await page.waitForSelector('#__next', {timeout: 120000});
hidden: true
means puppeteer will wait for the selector to disappear. This was the logic before: wait until cloudflare class disappears, but we could also just wait for the opensea selector to appear. But as said above, for me it did not work consistently.
from opensea-scraper.
ok cloudflare-scraper repo I tested does not work, it broke since cloudflare started to reload the page (which we also experience): JimmyLaurent/cloudflare-scraper#39 (comment)
Unfortunately the waiting for 2 mins somehow also does not work consistently. Cant figure out why for now. Not sure what else to do.
I have to say sometimes cloudflare is getting more rigid and its a temporary thing, so chances are opensea experiences a ton of traffic and thats why cloudflare is more rigid, but we'll have to see if this issue persist long term.
Sorry theres no fix for now :( if anyone has ideas please share!
Thank you for your quick responses. It seems that it works again without changing anything. I think you were right with the assumption that cloudflare changed something because opensea had a lot of traffic. I'll keep this issue open a few more days and I'll report to you if something changed again
from opensea-scraper.
Ah great, thanks for the report, its good to know that this issue happens. I know there is a solution to bypass cloudflare but its not that simple to do. They also change stuff a lot so its always a cat and mouse game to keep up with the changes.
Its good that it works again and please do report after you test over the next few days!
from opensea-scraper.
the problem is still there when i am using cloud service to run the puppeteer.
i see that we are already using the stealth plugin.. :(
cloudflare keeps waiting like forever.
from opensea-scraper.
Did you try using a proxy? If you ran this on a cloud service it's probably just blacklisted by opensea.
I didnt run into anymore problems on Mac. If I try to run the script on Linux (computer in the same network) it stops right here without any errors like it's frozen.
✅ === OpenseaScraper.rankings() ===
=== OpenseaScraper.rankings() ===
...fetching 1 pages (= top 100 collections)
...opening url: https://opensea.io/rankings?sortBy=one_day_volume
...🚧 waiting for cloudflare to resolve
...exposing helper functions through script tag
...scrolling to bottom and fetching collections.
I use the same code 1:1 on mac and It worked on Linux a week ago. I'm clueless
from opensea-scraper.
Which cloud service did you use? I am useing firebase functions (which basically is google cloud) and it works for me. If you want I can share my setup.
from opensea-scraper.
@SKreutz can you open another issue for the linux bug?
from opensea-scraper.
Which cloud service did you use? I am useing firebase functions (which basically is google cloud) and it works for me. If you want I can share my setup.
I'm using compute engine for that, perhaps i should try with firebase in that case
I havent tried using a proxy yet..
It would be great if you can share your setup with me @dcts
from opensea-scraper.
Did you try using a proxy? If you ran this on a cloud service it's probably just blacklisted by opensea.
I didnt run into anymore problems on Mac. If I try to run the script on Linux (computer in the same network) it stops right here without any errors like it's frozen.
✅ === OpenseaScraper.rankings() === === OpenseaScraper.rankings() === ...fetching 1 pages (= top 100 collections) ...opening url: https://opensea.io/rankings?sortBy=one_day_volume ...🚧 waiting for cloudflare to resolve ...exposing helper functions through script tag ...scrolling to bottom and fetching collections.
I use the same code 1:1 on mac and It worked on Linux a week ago. I'm clueless
Exactly, I took screenshots during the froze and it happens that the waiting page is being redirected to every time, so it is stuck within that waiting room forever.
from opensea-scraper.
I'm closing this issue since the cloudflare problem is solved and opening a new one
from opensea-scraper.
Related Issues (20)
- docs should warn that browserInstance requires stealth plugin HOT 6
- Code freezes when running `OpenseaScraper.rankings()` HOT 6
- [BUG] `offersByScrolling()` and `offersByScrollingByUrl()` not properly working HOT 44
- Not working when deployed (Heroku) HOT 7
- Not working when deployed (Google Cloud). TimeoutError: waiting for selector `.cf-browser-verification` to be hidden failed: timeout 30000ms exceeded HOT 4
- adding testnet HOT 12
- question HOT 1
- Reading from Activity page HOT 1
- [FEATURE REQUEST] scrape multiple pages of rankings HOT 4
- add support api key header for open sea HOT 5
- scrapping owner address from a collection HOT 2
- OpenseaScraper.rankings
- [BUG] Cannot get any result for floor price HOT 6
- [FEATURE REQUEST] scrape collection activity HOT 3
- [BUG] Missing price in result HOT 2
- [BUG] Cannot get price HOT 10
- offersByScrolling() and offersByScrollingByUrl() does not work HOT 1
- How to scrape more than top 100? HOT 1
- floorprice is null HOT 6
- Can't obtain any information from opensea due to cloudflare HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opensea-scraper.