balzer82 / immoscraper Goto Github PK
View Code? Open in Web Editor NEWImmoscout24.de scraper and data analytics
Immoscout24.de scraper and data analytics
tried some variations but never got it to work
would be very appreciated!
thanks
Bjoern
Thanks for making it available. FYI! Thanks again!
Probably since the relaunch is24 added a bot detection feature to it's website.
urllib2 and not even selenium works.
Any workaround ideas?
Hi balzer82,
thanks for your beautifully commented scraper and analysis notebooks :). They will be giving me a fair headstart for my own small project. Do you mind putting your work under some open licence, so I can just build on your notebooks!
There is a helpful guide at https://www.software.ac.uk/resources/guides/adopting-open-source-licence
But I always thought the open licence jungle is best mapped out in the license-chart by Robbie Morrisson:
Thanks,
Jonas
The url for the search request contains
Bundesland and Stadt
But this often does not work
Examples:
Bayern/Ebersberg
Bayern/Erding
Bayern/Landsberg
immoscout returns error 410
Do you have any hint what is wrong with these cities?
I just got started with Python as I was interessted in Immo data and wanted to produce my own scrapes. I found your instruction very helpful. My Scraper was running perfectly fine for a month. Thank you! But since the beginning of June I get the following error:
'NoneType' object is not subscriptable
The problems stems from looping through the scripts list and finding the script with the term 'IS24.resultList'
Line 60:
soup = BeautifulSoup(urlquery(url), 'html.parser')
scripts = soup.findAll('script')
for script in scripts:
#print script.text.strip()
if 'IS24.resultList' in script.text.strip():
s = script.string.split('\n')
the script.text.strip() command does no longer output any text. I have tried many variations, but don t get it to work. I am currently going through the bs4 documentation to seek alternatives. For me the problem came up after a spyder update. Though, I don t know whether this stands in any relation with the appearence of the problem.
Maybe you can help?
Since a couple of weeks I get the following error, without having changed anything...
Traceback (most recent call last):
File "", line 1, in
page_soup = BeautifulSoup(urllib.request.urlopen("https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-mieten").read(),"lxml")
File "C:\Users\Felix\Anaconda3\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\Felix\Anaconda3\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Users\Felix\Anaconda3\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\Felix\Anaconda3\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Users\Felix\Anaconda3\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Users\Felix\Anaconda3\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
HTTPError
Is this representing a loading error?
The URL is correct and works if implemented manually. Does this mean that Immoscout has implemented means to protect itself against web scrapers?
I have also tried ths script with implementing the changing browser profiles which aims to act like more natural querries. This has also not helped.
Thanks for your help!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.