Create a table that has ID codes and names for different geographic regions so that names are standardized across datasets.

I propose using ISO3 codes for country level (which corresponds with GADM ADM0 GID) and then the GADM GID for lower levels.

To start, we'll get ADM0 for all countries, and up to ADM2 for US, Canada, and China. I hope to update this to eventually be global.

I'd recommend changing the date format for time series CSVs to ISO standard yyyy-mm-dd.

For example, in https://github.com/CEIDatUGA/COVID-19-DATA/blob/master/US/US_wikipedia_cases_fatalities/read_UScases_wikipedia.R

This would work for now, until next year...

as.Date(paste(cases$Date, "2020"), "%b %d %Y")

Note, for many of our models, we start time on December 1, 2019. We also anticipate data in 2021 at least.

The filenames of these do not match conventions. These will be fixed in tonight's pull. But could mess folks up downstream if they are using these data. I don't know who is though so. They can probably figure it out!

We started collecting test data today. If we're collecting test data we might as well also collect the number of positive tests. Tests data will be much more informative if we also have positive tests

• the first table in this website https://globalhealth5050.org/covid19/.
• third table from DC's .xlsx file (https://dcgov.app.box.com/v/DCHealthStatisticsData)
• Utah sex/age group plot that we had looked at earlier https://coronavirus-dashboard.utah.gov/#demographics

Get a github action that works for the china-dxy scraping script. I had something running but then a merge issue popped up that may have broken it, so I'm just creating this issue so we can keep better track of it.

Can you add data from 3/21 at 7pm? Currently cases recorded between noon and 7pm are assigned to the next day. Thanks

Both deaths and case data for 3/24/20 at 7pm do not agree with the website as of 11:55pm. Particularly, deaths is obviously erroneous. Please correct.

• manually add html for May 29, 30, 31
• fix for June dates
• fix data pull breaks

I was going through the UScases_by_state_wikipedia data set. There are states with negative increases of confirmed cases on certain days. What does it mean to have a negative value? Are they errors?

I don't see where the "cumulative fatalities reported by country" in the old timeline spreadsheet (https://docs.google.com/spreadsheets/d/1SC28cM52m6s1gTJutpvFxadT9GGu-li90tsqkGuaM48/edit#gid=219891324) got ported over to the github -- am I missing it?

Related, did the "key events" tab can ported over.

Fine for some of this to stay in the spreadsheet, but it needs to be clear what is the definitive source

Please add some meta data to the read me. There is an empty template at the bottom of the readme that you can copy and fill in.

us-wiki autoupdate : Mon Apr 20 10:04:41 UTC 2020 failed to return well formed data.

Check if wiki table structure has changed, update script, and run manually.

Please add meta data - what are columns, particularly the ones that aren't identifiable state abbreviations?

• set up repo according to R datascience guidelines
• Add john's uga email as contact in README (this is for public contacts)
• make repo public

At the moment the code to scrape data from GA public health department is not working. The format of the webpage (https://dph.georgia.gov/covid-19-daily-status-report) has changed, and I can't seem to figure out how to extract the html. I tried to update the xpaths and Robbie has also tried to play with the css, but no success so far. The html page is really simple, so it is puzzling that we are not able to do it. I will continue to update the data manually, but if you are experienced on web scrapping any help would be greatly appreciated @arw36 @mvevans89 @allopole @rlrichards

This is so great! I think y'all are missing Decatur County, GA (at least in the time series), though.

@mvevans89 Wondering potential for autopulling the apple mobility from the https://www.apple.com/covid19/mobility website? Seems the link for button clicking is: "https://covid19-static.cdn-apple.com/covid19-mobility-data/2007HotfixDev58/v2/en-us/applemobilitytrends-2020-05-12.csv", so could set up to pull for file name to include yesterdays date?

I would save this in:
global/global_apple_covid_mobility_reports per the pull request

It looks like the previous links that we were using to download data are no longer working. There is now a way to download CSV's of the data, but it doesn't not include the number of tests. I've got something working so far for everything but the test data

• Add the least restrictive license compatible with source datasets and R

I pressed the wrong hotkey when I was merging the master updates into my local version. I think I might have overwritten changes to the repo. I suggest reverting to commit 70e5c08 by Liliana and then I can re-push the data I have locally. I would do this myself but 1) I've never actually reverted to a previous commit before (weird) and 2) it seems like that's an administrator function that random contributors shouldn't be doing.

Sorry!!!!!!

I saw an issue filed about this on the CovidTracking project -- this is worth doing, but would take settling on a set of buckets to map all data into, and building a per-state model that interpolates values for those buckets from the values given [or, for states w/ line lists, perhaps recalculating precisely for the shared buckets]

Then one could ask the outlier states to standardize the buckets they use in their own reporting.

Ping me if there is still interest in this / if you've found a source for this rebucketed data.

As of this morning, script is now retuning a table with some data misaligned. I'm having the same issue with different code accessing the same wikipedia table. something might have changed with the wiki table structure that's causing the rvest/xml2 code to return wrong results.

Appears to be the totals row.

Create a github action that will run the script US/US_wikipedia_cases_fatalities/read_UScases_wikipedia.R. This needs to run twice a day, morning and night (specific times not important)

There appears to be a merge conflict in the province-master.csv file. The HEAD notifier is on the second line.

Do we have a dataset for state fatalities? If not, can we scrape this from wikipedia too?

Is it possible to also include daily new cases and deaths in the GA data spreadsheet as columns?

Robbie- I'm double checking the google sheets to make sure we haven't missed anything. Is the data in global_cases_by_country and https://docs.google.com/spreadsheets/d/1SC28cM52m6s1gTJutpvFxadT9GGu-li90tsqkGuaM48/edit?usp=sharing sheet 2 the same. If so can you update the meta data to make it clear it was a switch from manual to scraping?