Comments (9)
Updating just the user-agent isn't the only way to detect the actual platform of the browser.
Consider utilising 'sec' overrides.
sec_ch_ua = '"Examplary Browser"; v="73", ";Not?A.Brand"; v="27"' # for example
options = Options()
options.add_argument(f'--sec-ch-ua={sec_ch_ua}')
from undetected-chromedriver.
In a datacenter (what AWS is), you are detected per definition
from undetected-chromedriver.
Thank you for your answers! I am still wondering what possibilities there are for the server not to be detected. I already use proxies.
from undetected-chromedriver.
Who is detecting you? They may detect your IP. They be doing TLS fingerprint which is happening more and more
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/123.0.0.0 Safari/537.36",
if this is your UA, then of course they are going to detect you, you are telling them you are using headless chrome
from undetected-chromedriver.
In general, you are right. But what surprises me is that I'm not detected with my computer, even though I use the same configs and proxies. My computer for comparison:
"request": {
"headers": {
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/123.0.0.0 Safari/537.36",
"sec-ch-ua": "\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\"Windows\""
}
My assumption is therefore that detecting Linux is the problem.
from undetected-chromedriver.
There's a thousand things they'd be flagging you for just Linux.
As @ usr741852 said, there's probably some level of fingerprinting at play, and if you're using a more typical base EC2 machine, that'd be the easiest to encounter blocks on.
Easy to test though, just spin up an instance with Windows on.
from undetected-chromedriver.
Sounds like a good idea. I'll let you know as soon as I have the results
from undetected-chromedriver.
I have now run the bot on an AWS Windows server. Unfortunately, I had the same experience, although I started it once with and once without headless. It probably has nothing to do with the operating system, but with the fact that these are EC2 instances. Is there a way to bypass this?
from undetected-chromedriver.
To better understand the problem, here is some context:
The bot scrapes the data of the particular matches. When I start the bot with my computer, it finds them all. If I start it with the server, the website returns a limited offer of all matches. So, I'm not blocked but only limited because the bot is detected.
from undetected-chromedriver.
Related Issues (20)
- [NoDriver] - get_position(), save_screenshot() may be incorrect
- NODRIVER - how can i connect remoter browser like gologin browser with nodriver package
- NODRIVER: how i can use nodriver to login and capture every https and cookies when login?? HOT 1
- [Nodriver] - no way to get the dimension of an element
- Changing language HOT 1
- "Enhanced ad privacy in Chrome" pop-up HOT 2
- [Nodriver] How to attach nodriver to another existing session/browser? HOT 2
- [NODRIVER] Exception: Failed to connect to browser HOT 3
- [nodriver] support add_experimental_option
- [nodriver] support how to get page_source and current_url HOT 4
- NODRIVER - block requests HOT 1
- how do i get style attribute of a div element?
- [nodriver] support how to set timezone?
- [nodriver] how can i use a socks5 proxy with authentication? HOT 3
- This version of ChromeDriver only supports Chrome version 125 Current browser version is 124.0.6367.29 HOT 3
- NODRIVER Memory leak HOT 8
- Exsist Java Version?
- [nodriver] support how to switch iframe?
- [nodriver] load profile HOT 2
- [nodriver] support how to set cookies? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from undetected-chromedriver.