This software tool allows to extract big collections of Twitter news-sharing users, their news tweets and the full data structure of the shared articles.
The application is automatic and self-powered so that it can be run for indefinitely long sessions.
- Python (>3.4.0) and pip
- MongoDB
- Twitter API keys
- (optional) Face++ keys
Clone the repository:
git clone https://github.com/DataSciencePolimi/NewsAnalyzer.git
Inside the project folder initialize a python environment
virtualenv newsanalyzer-env
Activate it
source newsanalyzer-env/bin/activate
The install the requirements
pip install -r requirements.txt
If you don't have yet, obtain Twitter API credentials
Open credential.json
and fill the values with your keys:
{
"consumer_key" : "<twitter API consumer key>",
"consumer_secret" : "<twitter API consumer secret>",
"access_token" : "<twitter API access token>",
"access_token_secret" : "<twitter API access token secret>",
"faceplus_key" : "<face++ key (optional)>",
"faceplus_secret" : "<face++ secret (optional)>"
}
- Download and install MongoDB
- Run command
mongod
to start a MongoDB server on localhost (may require priviledges) - Run
setup.py
script inside application folder
In order to start collecting users, tweets and articles your database need to contain at least one article entity to feed the recursive pipeline.
You can run utils/get_seeds.py
to get a set of initial seeds or you can download our pre-collected dataset.
Then run main_pipeline.py