Comments (5)
What happens generally is they detect unusual activity and redirect you to a captcha completion page. If you want to scrape you will need to write something to detect that and then manually complete the captcha or forward it through a completion service.
When I ran into this problem I had to manually loading pages from my script into my browser and saving them, then extracting data I needed from the saved HTML. Despite the large amount of request I made, I was able to access the pages. I don't know what information Google tracks that kept them from detecting when I did the same behavior manually...
Depending on the amount of data you are collecting it might be better to just use a service which will scrape data for you. There are services out there which have more sophisticated algorithms for avoiding detection, or even use captcha solving services.
from search-engines-scraper.
Thanks for bringing this to my attention. I don't know if changing User-Agent will be enough, but you could give it a try. Which search engine produces this error? If I remember correcly Google is the most "sensitive" one and usually increasing the delay (with engine._delay
) helps. Clearing cookies may help too. This library uses a requests.Session()
object for HTTP request, which persists cookies and other HTTP parameters. You could try setting a new client with engine._http_client = http_client.HttpClient(proxy=proxy)
. If that doesn't help then it's possible that the search engine is detecting us with URL or POST data parameters and unless it requires Js code execution, we may be able to solve it with engine._first_page()
and engine._next_page()
methods.
from search-engines-scraper.
Yes, I am using it with Google. Thanks for your guide, I will try to make changes according to your instructions. Let's see where it goes.
from search-engines-scraper.
I've noticed the ban usually occurs after 4 or 5 searches in a row which returned no results.
from search-engines-scraper.
I meet the problem,too.How to deal the problem?I try time.sleep(1) after search() and it works briefly.May raise an Exception in engine.py and restart the search work?Wish add an Exception.
from search-engines-scraper.
Related Issues (20)
- How to stop the search HOT 3
- Results appear different than the browser HOT 5
- is it possible to add baidu also? HOT 1
- google result has problem it can only give out single result HOT 1
- duplicate search results HOT 1
- -------- HOT 1
- Feature request Brave search engine HOT 4
- Unsupported or invalid CSS selector HOT 2
- Hi. can I get result of the News section of google engine ? and second question ; can I store result of each page in a json file? thanks HOT 2
- 404 SERP links HOT 4
- I have published this library to Pypi HOT 1
- Different results between the search engine scraper and google HOT 2
- Feature request; toggle console output for searches HOT 2
- DuckDuckGo scraper fails HOT 3
- How to use proxy server? HOT 2
- ERROR HTTP 429 HOT 2
- suggestion: installation through pip HOT 1
- Suggestion: a minimum_search_results return inside search() class HOT 1
- Bing() class always provides redirect links instead of the real one
- Google() class doesn't respond to number query pages HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from search-engines-scraper.