activeconclusion / covid19_mobility Goto Github PK

Thanks for providing the data! I noticed that since this morning, the raw CSV file with the Apple data leads to a 404. Would be very much appreciated if you could bring it back :) Thank you!

Hello friends, thank you for this repo. Do you have any plans to include tomtom traffic index? https://www.tomtom.com/en_gb/traffic-index/

https://www.waze.com/covid19

Google recently published a mobility report with time-series in CSV format. You can download it on their website.
That means there's no need for a PDF file parser anymore. Due to that, I plan to change the concept of this repository.
Here are my points that I propose to implement here:

0. Archive the PDF parser as part of the great history of this repository.
1. Automatically download to this repository all available files (including PDF) on Google and Apple sites. If there are no problems with Google reports, the Apple website parser needs to be rewritten, because my ad-hoc solution does not work, unfortunately.
2. Make one summary file from Google and Apple reports of the following structure:

3. Make a simple visualization app for this data (for example, using Bokeh library).

Feel free to offer your suggestions here.
Thank you!

Google added in a recent update two columns:

1. iso_3166_2_code
2. census_fips_code

I'm not sure that these attributes are useful, but some fixes needed to the script.

Also, Google added more regions.

Hi,

Thank you so much for writing these scrapers and providing this data. I noticed that the Waze data stopped updating in July. Although it was paused for a time, it seems like they're now providing data again if you look at their website. Is there any way to continue updating the scraper? Thanks!

Source: https://movement.uber.com/

Currently, I don't have an idea of how better to process this dataset, but it can be useful.

Hi,

Waze mobility data has not been updated for 10 days.

How can we fix this issue?

Cheers

As I predicted, today there are problems with updates on the Apple website. As reported on their website:

Data for May 11-12 is not available and will appear as blank columns in the data set.

In this repository, the raw report will be the same as on the site, but the preprocessed report will not be affected in any way.
I will monitor the situation in the future and hope that Apple will fix the issues as soon as possible.

Well-collected data and beautiful visualizations - it's definitely very nice. But analytics for just analytics is usually useless in practice. So, my main question is next: how is possible to make a real call to action for communities/governments based on this data?
Feel free to share here your thoughts, your insightful dashboards and visualisations or your current researches.
Thanks!

It's very sad, and unfortunately, I won't be able to fix it quickly because I'm currently sick with a coronavirus🤒. Suggestions/ideas on how to handle this situation are highly welcome.

Job: https://github.com/ActiveConclusion/COVID19_mobility/runs/6247139189?check_suite_focus=true

Today, Apple changed report, so data processing script currently doesn't work. I noticed two main updates:

1. Added country column
2. Added data for the US counties

I think that it's good to process on the same principle such as in Google Reports: create one report for the worldwide (w/o US counties) and second detailed for the US.
I hope to do it in the next 1-2 days😊

I've been trying to calculate total values for the US through state values. It is intuitively clear that the value of each state must be weighed with the number of the state population, so the formula for the total value has the following form:

This approach gave almost the same results as the total values in the report (difference less than 1%) except for the category "parks", and I can't give any logical explanation for this mismatch. My calculations in Google sheets here (see columns M:R).
What's wrong? Where is an error? Maybe, I made a somewhere simple mistake in calculations.

Hi there,
As you know, this repo had some troubles with the size quota. So the current solution for data updates should be optimized. I am in the progress to find a better solution, so data will be migrated to another cloud storage or will be split across several repos. I will inform you about all changes in this topic.

!!! IMPORTANT NOTE: Many of the forks of this repo are also uncomfortably large for Github, so Github support will contact the owners of those forks and ask them to remove that data from GitHub too. Please fork this repo in case you are really want to contribute something. If you want to save the link or support this repo please press star. Thank you for your understanding and apologise for the inconvenience caused.

UPD: Please don't fork this repo until the size issues are resolved.

hi guys, looks like Waze data stopped around 20.7.

Lately, Apple has been adding new cities/sub-regions to its report every day. Due to this, I have not been able to solve this problem automatically yet, so I have to add new entries to the matching table manually. I will write here about each fix to the matching table.

Job: https://github.com/ActiveConclusion/COVID19_mobility/runs/3820562081?check_suite_focus=true

From Google Community Mobility Reports website:

2020-09-22 20:45 GMT‎

We have temporarily suspended updates in some categories to prepare for an improvement to how we’ll compute retail, recreation and other categories. The upcoming update will ensure consistency in the way that the data is being reported. We are working to resume updates as soon as possible.

Hi all,

The WAZE Mobility data has not been updated for the past 17 days. Could you please have a look and update it if you can as I can see that there are some new data points available.

Thank you in advance,

Regards

In order to generate the report, it should not be:
'if filename in files_source' instead of 'if filename not in files_source':

Hi, first at all thank you very much for share this data! Is a very useful resource.
Today I have read the data in a PAndas DataFrame and I have encountered duplications in date however with different data.
As an example:
country region date retail and recreation workplaces
199877 Nicaragua Total 2020-02-15 6 4
199878 Nicaragua Total 2020-02-16 4 1
200039 Nicaragua Total 2020-02-15 9 2
200040 Nicaragua Total 2020-02-16 7 -1

Is the first time I see that, could be the original data corrupted?
Thanks in advance for your att.
Marce

Recently, Apple changed the report format (added more regions and columns), so currently processing their data doesn't work. I will try to fix this issue in 1-2 days.

Hello, I see missing data in dataset for Apple data on March 12th
dataset for Slovakia
https://github.com/ActiveConclusion/COVID19_mobility/raw/master/summary_reports/summary_report_countries.xlsx
it's all "null" that day, but days before and after are okay

Thanks so much for posting this! I'm hoping to track mobility by date... the graphs Google released clearly show trends by county by date. Do you know of any way to scrape or otherwise obtain that information? Thanks

hi team, we have last update 4 days ago, is it your automation working ok? why we don't have updates on this dataset please?

First of all, thank you for providing this data!
I am using your dataset in my research (merged Google and Apple data: https://github.com/ActiveConclusion/COVID19_mobility/blob/master/summary_reports/summary_report_regions.csv), but I don't know how correctly write a reference for it. I don't understand, what does it mean "ActiveConclusion"? Is it some organisation or a simple nickname? Could you elaborate, please, how is better to cite your dataset for publication?
Thank you very much!

I think it would be a good idea to try to catch someone from Apple and ask them to keep their API endpoint active (maybe even for a fee). But there is no guarantee that such an attempt will be successful.