A framework for Twitter data collection.

• core contains the code for data collection
• ui contains the code for the UI, it can be automatically deployed to gh-pages. you can also use the original UI client if you deployed your own core collection by changing the access URL to your own.

For a full description, please check the preliminary version of Twitter Watch's paper.

The current version merges the core and api containers but you can use the version in the original architecture by using the docker-compose-2.yml file and commenting the second line in the launcher.sh file (nohup python api/main.py > logs_flask.txt &).

The overall implementation is a sequential execution of groups of tasks until the scheduled (and parallel) tasks are reached.

This tool was used to collect data on the Portuguese Twittersphere, the following figure summarizes the amount of accounts and tweets it collected.

• cp example.env .env and edit
• docker-compose up (pass -d for detached mode)

docker exec some-mongo sh -c 'exec mongodump -d twitter --archive' > PATHTOLOCALFILE/dump.archive

docker exec -it twitter-watch_core_1 bash

Check pre-commit.com for more pre-commit functionality and then add it to the pre-commit config file.

To run, execute pre-commit run --all-files.