• This repo only contatins the data and statistics for 2022.For the data of:
  • - 2020 please visit:https://github.com/lopezbec/COVID19_Tweets_Dataset_2020
  • - 2021 please visit:https://github.com/lopezbec/COVID19_Tweets_Dataset_2021
• Data Organization
• Data Statistics
  • General Statistics
  • Language Statistics
  • English Sentiment Analaysis
  • English Named Entity Recognition, Mentions, and Hashtags
  • Spanish Sentiment Analaysis
  • Spanish Named Entity Recognition
  • Data Collection Process Inconsistencies
• Hydrating Tweets
  • Using our TWARC Notebook
    • Using Hydrator
    • Using Twarc
• Inquiries & Requests
• Licensing
• References

The repository contains an ongoing collection of tweets associated with the novel coronavirus COVID-19 since January 22nd, 2020.

As of 12/31/2022 there were a total of 3,001,855,651 tweets collected. The tweets are collected using Twitter’s trending topics and selected keywords. Moreover, the tweets from Chen et al. (2020) was used to supplement the dataset by hydrating non-duplicated tweets. These tweets are just a sample of all the tweets generated that are provided by Twitter, and it might not represent the whole population of tweets at any given point.

Citation

Lopez, C. E., Gallemore, C., “An Augmented Multilingual Twitter dataset for studying the COVID-19 infodemic” Soc. Netw. Anal. Min. 11, 102 (2021). DOI: s13278-021-00825-0 https://pubmed.ncbi.nlm.nih.gov/34697560/

The dataset is organized by hour (UTC) , month, and by tables. The description of all the features in all seven tables is provided below. For example, the path “./Summary_Details/2020_01/2020_01_22_00_Summary_Details.csv” contains all the summary details of the tweets collection on January 22nd at 00:00 UTC time.

For more information visit: Twitter API and the Documentation for API Tweet-object

The directory NYT_COVID_with_Reverse_Geo contains files in which Tweets with Geolocation are mapped to specific US state and county, alongside with the accumulative number of cases and death from the NY Time COVID-19 dataset. The tweets with geolocation information were ‘reverse geocode” using tidygeocoder, and Nominatim API. The tweets with geocoding information that were not able to be reverse geocode to a US state and county were excluded from this data.

As of 12/31/2022:

Total Number of tweets: 3,001,855,651

Average daily number of tweets: 115,932

There is a total of 6,729,323 tweets with geolocation information, which are shown on a map below:

The sentiment of all the English tweets was estimated using a state-or-the-art Twitter Sentiment algorithm BB_twtr. (See code here) .

The Named Entity Recognition algorithm of flairNLP was used to extract topics of conversation about PERSON, LOCATION, ORGANIZATION, and others. Below are the top 5 NER, Mentions (@) and Hastags (#)

The sentiment of all the Spanish tweets was estimated using sentiment analysis in spanish based on neural networks model of the the python library sentiment-analysis-spanish 0.0.25.

The Spanish Named Entity Recognition algorithm of flairNLP was used to extract topics of conversation about PERSON, LOCATION, ORGANIZATION, and others. Below are the top 5 NER of all the Spanish tweets (* some special character in Spanish are not correctly represented in the readme file, like character with accent mark)

The plots below show the normalized COVID-19 Cased vs the normalized number of geolocated Tweets for the top 2 most populated states and the top 2 least populated state:

Only tweets in English were collected from 22 January to 31 January 2020, after this time the algorithm collected tweets in all languages. There are also some known gaps of data shown below:

The notebook Automatically_Hydrate_TweetsIDs_COVID190_v2.ipynb will allow you to automatically hydrate the tweets-ID from our COVID19_Tweets_dataset GitHub repository.

You can run this notebook directly on the cloud using Google Colab (see how to tutorials) and Google Drive.

In order to hydrate the tweet-IDs using TWARC you need to create a Twitter Developer Account.

The Twitter API’s rate limits pose an issue to fetch data from tweed-IDs. So, we recommended using Hydrator to convert the list of tweed-IDs, into a CSV file containing all data and meta-data relating to the tweets. Hydrator also manages Twitter API Rate Limits for you.

For those who prefer a command-line interface over a GUI, we recommend using Twarc.

Follow the instructions on the Hydrator github repository.

Follow the instructions on the Twarc github repository.

If you would like to filter the tweets’ ID based on some metadata not provided on the repo (e.g., geolocation), if you would like to run some additional analyses on the full tweet text data (e.g., sentiment analysis using another language model, topic modeling, etc.), or if you have any questions about the dataset, please contact Dr. Christian Lopez at [email protected]

Existing filters performed are located in ‘Tweets_ID_Filter_requests’ directory

This dataset is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License (CC BY-NC-SA 4.0). By using this dataset, you agree to abide by the stipulations in the license, remain in compliance with Twitter’s Terms of Service, and cite the following manuscript:

Christian Lopez, and Caleb Gallemore (2020) An Augmented Multilingual Twitter Dataset for Studying the COVID-19 Infodemic. DOI: 10.21203/rs.3.rs-95721/v1 https://www.researchsquare.com/article/rs-95721/v1

Lopez, C. E., Gallemore, C., “An Augmented Multilingual Twitter dataset for studying the COVID-19 infodemic” Soc. Netw. Anal. Min. 11, 102 (2021). DOI: s13278-021-00825-0 https://pubmed.ncbi.nlm.nih.gov/34697560/

Emily Chen, Kristina Lerman, and Emilio Ferrara. 2020. #COVID-19: The First Public Coronavirus Twitter Dataset. arXiv:cs.SI/2003.07372, 2020

https://github.com/echen102/COVID-19-TweetIDs