Comments (8)
Yes, this module is supposed to scrape public data only. Now, we can't access pages without logging in which leads to a blocker
from twitter-scraper-selenium.
so does that mean we cant scrape data at all? without logging in ?
from twitter-scraper-selenium.
Yeah, without logging in it is not possible currently
from twitter-scraper-selenium.
Yeah, without logging in it is not possible currently
I how could we implement the logging in? Could you show how in code?
from twitter-scraper-selenium.
Yeah, without logging in it is not possible currently
I have logged in, still not working, what now?
from twitter-scraper-selenium.
So scraping tweets now works, although I'm getting random 10, not the latest 10. I'm using this code:
from twitter_scraper_selenium import scrape_profile
scrape_profile(twitter_username="tim_cook",output_format="csv",browser="firefox",tweets_count=10,filename="tim_cook_tweetsV3",directory="C:/Users/MCOBPHD19/OneDrive/Documents/Dissertation")
But scraping profile details doesn't. Using:
from twitter_scraper_selenium import get_profile_details
twitter_username = "tim_cook"
filename = "timCookProfile"
get_profile_details(twitter_username=twitter_username, filename=filename)
The error I get is this. Is this still the authentication issue? If so, why can I now scrape tweets?
TypeError Traceback (most recent call last)
C:\Users\MCOBPH~1\AppData\Local\Temp/ipykernel_3168/1681914123.py in
4 twitter_username = "tim_cook"
5 filename = "timCookProfile"
----> 6 get_profile_details(twitter_username=twitter_username, filename=filename)
~\anaconda3\lib\site-packages\twitter_scraper_selenium\profile_details.py in get_profile_details(twitter_username, proxy, filename, directory)
56 # if filename was not provided then print the JSON to console
57 return json.dumps(data)
---> 58 elif filename != '' and len(data) > 0:
59 # if filename was provided, save it to that file
60 mode = 'w'
TypeError: object of type 'NoneType' has no len()
from twitter-scraper-selenium.
So scraping tweets now works, although I'm getting random 10, not the latest 10. I'm using this code: from twitter_scraper_selenium import scrape_profile
scrape_profile(twitter_username="tim_cook",output_format="csv",browser="firefox",tweets_count=10,filename="tim_cook_tweetsV3",directory="C:/Users/MCOBPHD19/OneDrive/Documents/Dissertation")
But scraping profile details doesn't. Using: from twitter_scraper_selenium import get_profile_details
twitter_username = "tim_cook" filename = "timCookProfile" get_profile_details(twitter_username=twitter_username, filename=filename)
The error I get is this. Is this still the authentication issue? If so, why can I now scrape tweets? TypeError Traceback (most recent call last) C:\Users\MCOBPH~1\AppData\Local\Temp/ipykernel_3168/1681914123.py in 4 twitter_username = "tim_cook" 5 filename = "timCookProfile" ----> 6 get_profile_details(twitter_username=twitter_username, filename=filename)
~\anaconda3\lib\site-packages\twitter_scraper_selenium\profile_details.py in get_profile_details(twitter_username, proxy, filename, directory) 56 # if filename was not provided then print the JSON to console 57 return json.dumps(data) ---> 58 elif filename != '' and len(data) > 0: 59 # if filename was provided, save it to that file 60 mode = 'w'
TypeError: object of type 'NoneType' has no len()
I'm struggling with this same issue. Is there any workaround?
from twitter-scraper-selenium.
There are still problems for users scraping without logging in. The tweets are not present the same as it is present for logged-in users on Twitter
from twitter-scraper-selenium.
Related Issues (20)
- Unable to launch the lib HOT 2
- Proxy HOT 3
- How to show progress for the scraping process? Right now there is no indication that it works until it is all done.
- AttributeError: 'str' object has no attribute 'close' HOT 10
- More Examples and Documentation Needed HOT 1
- API rate limit exceeded with scrape_profile() function HOT 5
- login support before executing apis HOT 3
- Install help: selenium.common.exceptions.SessionNotCreatedException HOT 1
- why the very long wait in wait_until_completion( ) HOT 2
- scrape_keyword : search in array of keyword not just a string HOT 1
- The address returned by calling scrape_profile is wrong!
- not enough values to. unpack HOT 1
- timeout exception HOT 3
- Tweets did not appear HOT 7
- Failed to make request! HOT 1
- Tweets not ordered by date HOT 2
- Inject javascript to get media url
- twitter_scraper_selenium.scraping_utilities:Error at find_x_guest_token: 'guest_token'
- add input tweet url to scrape tweet comments and nested replies
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from twitter-scraper-selenium.