Covid19_Analysis
NOTE It is recommended that you view this website via a PC or Laptop as some features may not be visible properly on Mobile devices.
This App is not hosted yet, it can be used via this Ngrok link: ____ (this can be changed everytime, and it may not be available at odd hours as it is locally hosted for demo purpose) Check screenshots below to see the actual APP.
All the data has been collection from official Covid-19 resource:
- https://api.covid19india.org/
- https://apisetu.gov.in/public/api/cowin (use with caution: can be rate limited. I have added caching to avoid hitting the API endpoint every call)
- https://twitter.com/home
Pre-requisites:
-
Create Twitter API tokens
- Create an account on
https://developer.twitter.com
- Create an application with a unique name
- To access the Twitter API, you will need 4 things from the your Twitter App page. These keys are located in your Twitter app settings in the Keys and Access Tokens tab.
- consumer key
- consumer secret key
- access token key
- access token secret key
- Create a directory called
creds
and a filecredentials.py
inside it. Save these credentials as Variables in it - Do not share these with anyone else because these values are specific to your app.
- Create an account on
-
Docker for Windows/Linux
- Link Here -> https://docs.docker.com/docker-for-windows/install/
- 4GB RAM
- Windows Subsystem for Linux (for Windows users)
-
MongoDB, MongoDB (Compass)
- Link Here -> https://www.mongodb.com/try/download/community
- Setup process available on the site
-
Redis, redis-scheduler For Windows:
- Need Windows Subsystem for Linux (WSL) - Ubuntu (
https://ubuntu.com/wsl
) - Inside the Ubuntu Terminal,
sudo service redis-server start
- To test,
redis-cli
->ping
-> Expected Response:PONG
- Redis is working
For Linux:
- apt install redis
- setup docs available online for Linux
- Need Windows Subsystem for Linux (WSL) - Ubuntu (
Steps to Run:
- Clone the repository
https://github.com/eeshsingh123/Covid19_Analysis.git
- Setup Docker & docker-compose in the system
- Run the command,
docker-compose up --build -d
(-d for daemon build)
Steps to Run (Without Docker):
- Terminal 1 : python3 -m app.flask_main (
to run flask app
) - Terminal 2 : python3 -m app.twitter_main (
to run twitter app
) - Terminal 3: python3 -m rq_queue data_updater (
run the redis queue, to update the reports based on scheduler call
) - Terminal 4: rqscheduler -v (
Scheduler which runs periodically and collects users, hashtags data from twitter and updates graph data from covid api
)
Repository Contents:
-
/app/flask_main.py
: Flask app which handles the API calls and interacts with the JS Front-end code using Jinja2 template -
/app/twitter_main.py
: Runs 24/7 and collects data from the Twitter Stream leveraging the tweepy stream api, stores data into mongoDB and generates trending on a Runtime -
/workers
: Consists of the API and Twitter data Aggregation, Preprocessing and Modelling logic before sending it to the Front-end via Flask app -
/concepts
: Consists of various Jupyter notebooks where each logic has been tried and tested before integration -
/static & /templates
: Has all the JavaScript, CSS & Templates file used by Flask for UI integration -
/tools
: Contains Helper functions to perform tasks needed