Comments (11)
Yes, you are already telling the reason 😂. Sadly cloudflare protection is not possible to circumvent. Also thought there was a ticket for this already. Looks like not.
I have no idea how this should be possible. The only thing is launching a real browser by Open Directory Downloader, that the user does the cloudflare part, and 'snoop' the cookie which will be used to scan. But that is no easy task.
So for to foreseeable future this is not possible. :(
from opendirectorydownloader.
The only thing is launching a real browser by Open Directory Downloader, that the user does the cloudflare part, and 'snoop' the cookie which will be used to scan.
I implemented this a while back I think, passing the cookie to ODD in order to index a CF-protected OD.
I could dig that out and add some finishing touches along with a small guide, if you guys want that?
from opendirectorydownloader.
That would be great, but you cannot add the cookie to ODD? Or am I missing something? 😂
from opendirectorydownloader.
but you cannot add the cookie to ODD?
that's the part I implemented 😉
from opendirectorydownloader.
Hmm, but not part of the code here? Still confused :P
from opendirectorydownloader.
Yeah, I never committed it because it wasn't finished yet. I just added a constant somewhere in the code.
Shouldn't be too hard to expose it as a flag though :)
from opendirectorydownloader.
It was still on the list (your own ticket) 😂
from opendirectorydownloader.
It was still on the list (your own ticket) 😂
🤦🏻♂️
Guess I'll get to work then 😁
from opendirectorydownloader.
I got to work too, first version is finished. It's not that reliable, not sure if the site/cloudflare is the issue, or cloudflare doing it on purpose. It cannot be tested yet. No build created yet. First need to improve it, because I see 403 Forbidden responses when the cloudflare part was still working, and popping up browsers every second, or sometimes looks like even faster than that. On that point, you need to quickly close the indexer 😂
https://github.com/KoalaBear84/OpenDirectoryDownloader/tree/browser
Can't promise anything.
from opendirectorydownloader.
Merged PR #108 which includes experimental Cloudflare support.
Be sure to read the readme! It does work, but maybe when using multiple threads, or long scans, it will fail.
Let me know how it works and if you run into any problems.
If anybody have more Cloudflare protected sites to test with, let me know.
https://github.com/KoalaBear84/OpenDirectoryDownloader/releases/tag/v2.2.0.1
from opendirectorydownloader.
@itoshikisetnom You can test it out. It's not 100% in my test, but it could work reasonable.
from opendirectorydownloader.
Related Issues (20)
- Support for http authentication HOT 5
- No URLs to save for AliceAndBooks.com HOT 6
- Crash when not specifying a protocol HOT 5
- No folder for LogArchives on systems with `/` as path seperator HOT 4
- SIGABRT on unknown cli args HOT 4
- Add version to JSON
- Add removal URL for uploaded files to JSON HOT 1
- Add mounting support HOT 5
- Actual OD Downloading support HOT 1
- Continous Downloading/Crawling depending on Timestamp HOT 1
- Copyparty `--force-js` support HOT 3
- Unsupported site HOT 1
- S3 support HOT 4
- 2.9.0.3 errors HOT 4
- Only single Chromium instance
- cant open in colab, giving error HOT 1
- JSON save error when using custom port HOT 1
- ways to bypass Cloudflare HOT 1
- How to Paste URLs into the program. HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opendirectorydownloader.