Cloudflare DDOS protection about undetected-chromedriver HOT 46 CLOSED

from undetected-chromedriver.

lol.
Seemed the issue was probably due to character repetition, and the package is updated to 1.5.1 with a minor change.

Note: i did not implement the random function proposed and PR'd by @czoins (yet) . This is on purpose, and has other reasons then most of you would probably think of. However please let me know the second this issue returns.

Meanwhile In the background i'm still busy rewriting and testing a new monstrous function on all platforms (without prying eyes), which will be able to bypass even future enhancements. Thanks @czoins for your work.

from undetected-chromedriver.

Simply replace the "undetectedchromeDRiver" with something with equivalent length. Worked for me.

from undetected-chromedriver.

If it helps, you can use Playwright (Python C#) instead, and choose Firefox or WebKit browser (Chromium somehow can still be detected).
It's similar to selenium and it can bypass Cloudflare protection even in headless mode, I've tried python and C# codes on SteamDB (which also uses CF protection), it works.

from undetected-chromedriver.

Simply replace the "undetectedchromeDRiver" with something with equivalent length. Worked for me.

undetected-chromedriver/undetected_chromedriver/__init__.py

Line 233 in 7400dd4

newline = re.sub(b"cdc_.{22}", b"xxx_undetectedchromeDRiver", line)

Which would mean Cloudflare is checking my little repo for their antibot algo, that's pretty genius as well.
I assume you deleted your chromedriver executable after making the change so it patches it with your random input?
That would be a simple change to implement if that really works, however i've tested this is as well, and did not have positive results. please reply if it's working for any of you.

I have edited the script so it randomizes the string each time it executes. So yes, in another words i have bascially deleted the chromdriver. I can confirm that it works, as when i placed back the old string it no longer does work. Whether this works for all cloudflare protected sites isn't clear, but don't they all use the same algorithm?

from undetected-chromedriver.

@ultrafunkamsterdam it's working for me. thanks @czoins

from undetected-chromedriver.

Related to #38 and means that it either took CF more than a month, to update their code to detect this, or they just found this repo and joined the "watch party" recently :)

from undetected-chromedriver.

I did not create this project to forward poeple to other repo's. Please stop "plugging" these solutions. It is not a solution to copy a cookie between browsers using 2 libraries of jou could also just solve the damn captcha in 2 seconds and be rid of it. I just finished draft, and will upload a video demonstrating it goes to several heavy protected hcaptcha/cloudflare/custom impl. sites. Give me another 2 days and the package will be updated.

from undetected-chromedriver.

Simply replace the "undetectedchromeDRiver" with something with equivalent length. Worked for me.

undetected-chromedriver/undetected_chromedriver/__init__.py

Line 233 in 7400dd4

newline = re.sub(b"cdc_.{22}", b"xxx_undetectedchromeDRiver", line)

@czoins You're fucking genius, thanks man ✌

from undetected-chromedriver.

Simply replace the "undetectedchromeDRiver" with something with equivalent length. Worked for me.

undetected-chromedriver/undetected_chromedriver/__init__.py

Line 233 in 7400dd4

newline = re.sub(b"cdc_.{22}", b"xxx_undetectedchromeDRiver", line)

Which would mean Cloudflare is checking my little repo for their antibot algo, that's pretty genius as well.
I assume you deleted your chromedriver executable after making the change so it patches it with your random input?
That would be a simple change to implement if that really works, however i've tested this is as well, and did not have positive results. please reply if it's working for any of you.

from undetected-chromedriver.

I am having the same issue. Have you found a solution yet by any chance?

from undetected-chromedriver.

https://www.mc-market.org same issue

from undetected-chromedriver.

That's great to hear! Would you mind sharing what it was that was causing the issue?

from undetected-chromedriver.

I have the same issue !

from undetected-chromedriver.

Same issue too ):

from undetected-chromedriver.

Related to #38 and means that it either took CF more than a month, to update their code to detect this, or they just found this repo and joined the "watch party" recently :)

he predicted this happening lmao 😳

from undetected-chromedriver.

I think I have the same issue with tirexo.cc

@ultrafunkamsterdam, Let me know if I can assist you for the rewrite by testing or otherwise

from undetected-chromedriver.

Cloudflare seems to detect it again, even with randomized 'undetectedchromeDRiver'. You can test on https://v3rmillion.net, https://www.sythe.org, https://www.mc-market.org

If I can do anything to help, do let me know..

from undetected-chromedriver.

Same here, cloudflare detection started 2 days ago for me.

from undetected-chromedriver.

Same here as well. Is there any workaround to the new Cloudflare detection?

from undetected-chromedriver.

Cloudflare still detect checking :(

from undetected-chromedriver.

If it helps, you can use Playwright (Python C#) instead, and choose Firefox or WebKit browser (Chromium somehow can still be detected).
It's similar to selenium and it can bypass Cloudflare protection even in headless mode, I've tried python and C# codes on SteamDB (which also uses CF protection), it works.

Works great.
If you want to continue working with selenium you can just grab the cookies fom playwight context after visiting the site and add them to the driver instance.

from undetected-chromedriver.

Somewhat of an ignorant question, but I assume youd want to grab the cookies if you're working with a form of authenticated session? Any other reasoning or benefits to that just as it pertains to avoiding bot detection? @bl4ckOut

from undetected-chromedriver.

I just want to get past the cloudflare protection. After you successfully loaded a site with playwright the corresponding cloudflare cookies (like 'cf_clearance' and '__cfduid') have been authenticated and can be used in other sessions. There are no other advantages. I just like selenium and don't want to learn a new scraper.

from undetected-chromedriver.

Ah makes sense. Thanks for that, I wasnt aware that cloudflare stores those cookies.

from undetected-chromedriver.

As a workaround until the issue is fixed, you can automate chrome itself which isn't detected. I am not sure if you can run multiple instances though.

import subprocess
#launch chrome with remote debug, and open your url (you can also use a separate user directory (--user-data-dir) which is advisable)
subprocess.Popen(['C:/Program Files (x86)/Google/Chrome/Application/chrome.exe', '--remote-debugging-port=7102', 'url'])
#might want to add a time.sleep here until chrome launches
chrome_options.add_experimental_option("debuggerAddress","localhost:7102")
driver = uc.Chrome(options=chrome_options)
#connected, use the same way as chromedriver
...

from undetected-chromedriver.

@czoins although i like the approach ( caugh ... it's how new version is implemented), and it works without patching as well, it does get detected by cf IUAM mode for example. Do you, by chance, have a working setup in headless mode. I'm running against a huge lock and memory builds FAST until 30G is used so i wonder if you got it working for headless.

from undetected-chromedriver.

I know this is likely outside the scope of the library given its more about how it’s used, but at least 1 notable scraping repo seems to enforce a wait after a page loads when it detects the host is using cloudflare. I’m not sure how that could be incorporated here, but is anyone familiar with what detection that’s attempting to circumvent? Maybe there something that could help.

from undetected-chromedriver.

from undetected-chromedriver.

If it helps, you can use Playwright (Python C#) instead, and choose Firefox or WebKit browser (Chromium somehow can still be detected).
It's similar to selenium and it can bypass Cloudflare protection even in headless mode, I've tried python and C# codes on SteamDB (which also uses CF protection), it works.

Playwright works as a charm.
Goodbye selenium.

from undetected-chromedriver.

to test:

pip install -U undetected_chromedriver

import undetected_chromedriver.v2 as uc

driver = uc.Chrome()
with driver:
    driver.get('https://lhcdn.botprotect.io')

from undetected-chromedriver.

I updated through pip as per your instructions but importing undetected_chromedriver.v2 fails

from undetected-chromedriver.

@jjstecchino, pip does not update to beta versions. I did not know that, so for simplicity i released it, and now it should work.

from undetected-chromedriver.

Thanks for doing that. I was able to get it. Now on executing

driver = uc.Chrome()

I get an error:

Exception has occurred: TypeError
expected str, bytes or os.PathLike object, not NoneType

I am on a Mac. chromedriver gets downloaded in my test project directory.

from undetected-chromedriver.

trying to debug, it seems in v2.py find_chrome_executable() can't find chrome executable on osx (using Big Sur)

from undetected-chromedriver.

Still getting blocked by Cloudflare in v2 :(

from undetected-chromedriver.

Still getting blocked by Cloudflare in v2 :(

I fixed a few things on the v2 driver to find chrome executable on osx, but I can now go past Cloudflare

from undetected-chromedriver.

@MjedAl

please try and close every chrome or chromedriver process, before starting.

import undetected_chromedriver.v2 as uc
driver = uc.Chrome()

with driver:
    driver.get("https://coinfaucet.eu")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("coinfaucet.eu.png")

with driver:
    driver.get("https://cia.gov")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("cia.gov.png")

with driver:
    driver.get("https://lhcdn.botprotect.io")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("notprotect.io.png")

with driver:
    driver.get("https://www.datadome.co")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("datadome.co.png")

@jjstecchino , bummer, i tested it on Catalina, which worked, i'll fix that asap.

from undetected-chromedriver.

@MjedAl

please try and close every chrome or chromedriver process, before starting.

import undetected_chromedriver.v2 as uc
driver = uc.Chrome()

with driver:
    driver.get("https://coinfaucet.eu")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("coinfaucet.eu.png")

with driver:
    driver.get("https://cia.gov")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("cia.gov.png")

with driver:
    driver.get("https://lhcdn.botprotect.io")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("notprotect.io.png")

with driver:
    driver.get("https://www.datadome.co")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("datadome.co.png")

@jjstecchino , bummer, i tested it on Catalina, which worked, i'll fix that asap.

Because of the import .v2 deletes all files in the project directory
https://youtu.be/8o02waJqd1k

from undetected-chromedriver.

@ultrafunkamsterdam any ideas why/how CF detects chromedriver and why with remote debug it doesn't?

from undetected-chromedriver.

Hi! How can use the patch when using a selenium standalone (or hub) docker image, so using the Remote webdriver?

from undetected-chromedriver.

@ultrafunkamsterdam any ideas why/how CF detects chromedriver and why with remote debug it doesn't?

@chemeng , likely because chrome is started within the chromedriver process (which makes it a direct fork/child of chromedriver), thus sharing environments, and using remote debugging you start both chrome and chromedriver as seperate processes under your user. This also happens to enable the functionality to disconnect and reconnect without closing the browser window. This interferes the CF check which uses the samesite=LAX cookie, https://support.cloudflare.com/hc/en-us/articles/360038470312-Understanding-SameSite-cookie-interaction-with-Cloudflare )

from undetected-chromedriver.

@MjedAl
please try and close every chrome or chromedriver process, before starting.

import undetected_chromedriver.v2 as uc
driver = uc.Chrome()

with driver:
    driver.get("https://coinfaucet.eu")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("coinfaucet.eu.png")

with driver:
    driver.get("https://cia.gov")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("cia.gov.png")

with driver:
    driver.get("https://lhcdn.botprotect.io")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("notprotect.io.png")

with driver:
    driver.get("https://www.datadome.co")
time.sleep(4) # sleep only used for timing of screenshot
driver.save_screenshot("datadome.co.png")

@jjstecchino , bummer, i tested it on Catalina, which worked, i'll fix that asap.

Because of the import .v2 deletes all files in the project directory
https://youtu.be/8o02waJqd1k

Same here.

from undetected-chromedriver.

having the same problem, anyone have found a solution ?

from undetected-chromedriver.

Closed. Open a new issue with code and error log if you think you are having this issue and you are 100% sure your ip is clean.

from undetected-chromedriver.

Simply replace the "undetectedchromeDRiver" with something with equivalent length. Worked for me.

undetected-chromedriver/undetected_chromedriver/__init__.py

Line 233 in 7400dd4

newline = re.sub(b"cdc_.{22}", b"xxx_undetectedchromeDRiver", line)

Can you please check if you find also a solution for trainline.com, apparently after 50 rounds of chromedriver, they still somehow manage to profile the driver and block it, with stormproxies

from undetected-chromedriver.