Code Monkey home page Code Monkey logo

covid19_mobility's Introduction

Buy Me A Coffee

COVID-19 Mobility Data Aggregator. Scraper of Google, Apple, Waze and TomTom COVID-19 Mobility Reports

This is a repository with a data scraper of Mobility Reports and reports in different formats.

Data reports are no longer updated.

Table of contents

  1. About data
  2. Data explorer
  3. How to run script
  4. Contributing
  5. Showcases

About data

In early April 2020, Google started publishing an early release of COVID-19 Community Mobility Reports to provide insights into what has changed in response to work from home, shelter in place, and other policies aimed at flattening the curve of this pandemic. These reports have been developed to be helpful while adhering to our stringent privacy protocols and policies.

These Community Mobility Reports aim to provide insights into what has changed in response to policies aimed at combating COVID-19. The reports chart movement trends over time by geography, across different categories of places such as retail and recreation, groceries and pharmacies, parks, transit stations, workplaces, and residential.

Update interval: The Community Mobility Reports are no longer being updated as of 2022-10-15.

By downloading or using this data and reports, you agree to Google Terms of Service.

The CSV file shows a relative volume of directions requests per country/region or city compared to a baseline volume on January 13th, 2020.

Day defined as midnight-to-midnight, Pacific time. Cities represent usage in greater metropolitan areas and are stably defined during this period. In many countries/regions and cities, relative volume has increased since January 13th, consistent with normal, seasonal usage of Apple Maps. Day of week effects are important to normalize as you use this data.

Data that is sent from users' devices to the Maps service is associated with random, rotating identifiers so Apple doesn't have a profile of your movements and searches. Apple Maps has no demographic information about Apple users, so it's impossible to make any statements about the representativeness of usage against the overall population.

Update interval: As of April 14, 2022, Apple is no longer providing COVID-19 mobility trends reports.

By downloading or using this data, you agree to Apple terms.

The driven kilometers/miles percent change data being shared comes from the Waze app and is aggregated and anonymized. These insights were generated using differential privacy to protect user privacy. No personally identifiable information, such as an individual’s location, contacts, or movement, is available through this data.

These reports show the increase or decrease in driven kilometers/miles as a percent change compared to a baseline. The changes for each day are compared to a baseline value for that day of the week.

  • The baseline is the average value, for the corresponding day of the week, during the 2- week period February 11, 2020 to February 25, 2020.
  • The reports show trends over two weeks with the most recent data representing approximately 2-3 days ago.

As with all samples, this may or may not represent the exact behavior of a wider population.

Update interval: Waze dashboard has been retired and will no longer be updated as of July 2022

Covering 416 cities across 57 countries on 6 continents, Traffic Index ranks urban congestion worldwide and provides free access to city-by-city information. Wondering how we determine the rankings, and what exactly the percentages mean?

A 53% congestion level in Bangkok, for example, means that a 30-minute trip will take 53% more time than it would during Bangkok’s baseline uncongested conditions.

You can turn this 53% into travel time through simple calculations. First: 0.53 x 30 mins = 15.9 mins extra average travel time. Second: 30 mins + 15.9 mins = 45.9 mins total average travel time.

We calculate the baseline per city by analyzing free-flow travel times of all vehicles on the entire road network – recorded 24/7, 365 days a year. This information allows us to also calculate, for example, how much extra time a driver will spend in traffic during rush hour in Bangkok.

We perform calculations for all hours of each day, so you can see congestion levels at any time in any city, including morning and evening peak hours.

Update interval: daily

Data explorer

Google reports:

Raw CSV file (in ZIP archive). Direct link to the original CSV: https://www.gstatic.com/covid19/mobility/Global_Mobility_Report.csv

Data for the worldwide (only 1st level of subregions): CSV, Excel

Detailed reports:

Data for the US: CSV, Excel

Data for Brazil: CSV, Excel

Data for Europe: CSV (in ZIP archive), Excel

Data for Asia + Africa: CSV, Excel

Data for North and South America + Oceania (Brazil and US excluded): CSV, Excel

Apple reports:

Raw CSV file

Data for the worldwide: CSV, Excel

Data for the US: CSV, Excel

The following transformations have been made here:

  • transformed dates from columns to rows
  • transformed transportation types from rows to columns
  • subtracted 100 from values (such as in Google Mobility Reports)

Note: Data for May 11-12, 2020 and March 12, 2021 is not available

Waze reports:

Raw CSV files: Country-level, City-level

Preprocessed report: Google Sheets, CSV, Excel

TomTom reports:

Scraped reports: Google Sheets, CSV, Excel

diffRatio attribute shows relative difference of average congestion levels in 2020 from standard congestion levels in 2019.

Note: Data for China only available up to 21.02.2021

Summary reports:

These are merged Apple and Google reports.

Report by regions: CSV, Excel

Report by countries: Google Sheets, CSV, Excel

Report for the US: CSV, Excel

How to run script

Installation

git clone https://github.com/ActiveConclusion/COVID19_mobility
pip install -r requirements.txt

Usage

# scrape data from specified sources. If no sources are provided, data will be scraped from all available sources
python scraper.py scrape <SOURCES>

# merge mobility reports (Apple and Google)
python scraper.py merge

# Scrape data from all sources and merge reports
python scraper.py run-all

Also, available Jupyter notebook mirror of this script

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Place to discuss use cases for this data

Showcases

Dashboards and visualizations based on these data

  1. Dashboard for the US-1
  2. Dashboard for the US-2
  3. Dashboard for the world
  4. Balefire COVID-19 USA Data Explorer
  5. Pandemic Traffic in Ireland by David ó Cinnéide
  6. New South Wales COVID Tracking Dashboard by Damjan Vlastelica
  7. Global COVID Vital Signs
  8. Toronto After The First Wave. Mobility Dashboard
  9. [Here can be your great dashboard/visualization]

Articles and research publications

  1. Is Your Community Doing Enough To Fight COVID-19? by Molly Liebeskind
  2. Project US Mobility and Fuel Demand Under COVID-19
  3. COVID-19: Country progress tracker and forward projections
  4. Krekel, C., Swanke, S., De Neve, J., & Fancourt, D. (2020). Are Happier People More Compliant? Global Evidence From Three Large-Scale Surveys During Covid-19 Lockdowns
  5. Green shoots and mobility: Philippine economic prospects By Diwa C. Guinigundo
  6. Franks J, Gruss B, Mulas-Granados C, Patnam M, Weber S. Reopening strategies, mobility and COVID-19 infections in Europe: panel data analysis. BMJ Open. 2022;12(2):e055938. Published 2022 Feb 9. doi:10.1136/bmjopen-2021-055938
  7. Anna Godøy, Maja Weemes Grøtting. Implementation and economic effects of local non-pharmaceutical interventions. medRxiv 2022.02.10.22270783; doi:https://doi.org/10.1101/2022.02.10.22270783
  8. Strzelecki, A.; Azevedo, A.; Rizun, M.; Rutecka, P.; Zagała, K.; Cicha, K.; Albuquerque, A. Human Mobility Restrictions and COVID-19 Infection Rates: Analysis of Mobility Data and Coronavirus Spread in Poland and Portugal. Int. J. Environ. Res. Public Health 2022, 19, 14455. https://doi.org/10.3390/ijerph192114455
  9. Bublyk, M., Feshchyn, V., Bekirova, L., & Khomuliak, O. (2022). Sustainable Development by a Statistical Analysis of Country Rankings by the Population Happiness Level. COLINS.
  10. [Here can be your great article/research publication/paper]

covid19_mobility's People

Contributors

activeconclusion avatar dependabot[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

covid19_mobility's Issues

Scraper of Mobility reports v 2.0

Google recently published a mobility report with time-series in CSV format. You can download it on their website.
That means there's no need for a PDF file parser anymore. Due to that, I plan to change the concept of this repository.
Here are my points that I propose to implement here:

  1. Archive the PDF parser as part of the great history of this repository.
  2. Automatically download to this repository all available files (including PDF) on Google and Apple sites. If there are no problems with Google reports, the Apple website parser needs to be rewritten, because my ad-hoc solution does not work, unfortunately.
  3. Make one summary file from Google and Apple reports of the following structure:
country sub_region_1 sub_region_2 date retail grocery_and_pharmacy parks transit_stations workplaces residential walking driving transit
... ... ... ... ... ... ... ... ... ... ... ... ...
  1. Make a simple visualization app for this data (for example, using Bokeh library).

Feel free to offer your suggestions here.
Thank you!

Error in Google Reports?

I've been trying to calculate total values for the US through state values. It is intuitively clear that the value of each state must be weighed with the number of the state population, so the formula for the total value has the following form:
equation
This approach gave almost the same results as the total values in the report (difference less than 1%) except for the category "parks", and I can't give any logical explanation for this mismatch. My calculations in Google sheets here (see columns M:R).
What's wrong? Where is an error? Maybe, I made a somewhere simple mistake in calculations.

Why do we need this data?

Well-collected data and beautiful visualizations - it's definitely very nice. But analytics for just analytics is usually useless in practice. So, my main question is next: how is possible to make a real call to action for communities/governments based on this data?
Feel free to share here your thoughts, your insightful dashboards and visualisations or your current researches.
Thanks!

Apple: report format changed

Today, Apple changed report, so data processing script currently doesn't work. I noticed two main updates:

  1. Added country column
  2. Added data for the US counties

I think that it's good to process on the same principle such as in Google Reports: create one report for the worldwide (w/o US counties) and second detailed for the US.
I hope to do it in the next 1-2 days😊

[IMPORTANT] Data migration and optimization

Hi there,
As you know, this repo had some troubles with the size quota. So the current solution for data updates should be optimized. I am in the progress to find a better solution, so data will be migrated to another cloud storage or will be split across several repos. I will inform you about all changes in this topic.

!!! IMPORTANT NOTE: Many of the forks of this repo are also uncomfortably large for Github, so Github support will contact the owners of those forks and ask them to remove that data from GitHub too. Please fork this repo in case you are really want to contribute something. If you want to save the link or support this repo please press star. Thank you for your understanding and apologise for the inconvenience caused.

UPD: Please don't fork this repo until the size issues are resolved.

WAZE Mobility DATA not updated

Hi all,

The WAZE Mobility data has not been updated for the past 17 days. Could you please have a look and update it if you can as I can see that there are some new data points available.

Thank you in advance,

Regards

Waze Data Has Returned

Hi,

Thank you so much for writing these scrapers and providing this data. I noticed that the Waze data stopped updating in July. Although it was paused for a time, it seems like they're now providing data again if you look at their website. Is there any way to continue updating the scraper? Thanks!

Google: report format changed

Google added in a recent update two columns:

  1. iso_3166_2_code
  2. census_fips_code

I'm not sure that these attributes are useful, but some fixes needed to the script.

Also, Google added more regions.

Data with dates duplicated?

Hi, first at all thank you very much for share this data! Is a very useful resource.
Today I have read the data in a PAndas DataFrame and I have encountered duplications in date however with different data.
As an example:
country region date retail and recreation workplaces
199877 Nicaragua Total 2020-02-15 6 4
199878 Nicaragua Total 2020-02-16 4 1
200039 Nicaragua Total 2020-02-15 9 2
200040 Nicaragua Total 2020-02-16 7 -1

Is the first time I see that, could be the original data corrupted?
Thanks in advance for your att.
Marce

How correctly to cite your dataset for publication?

First of all, thank you for providing this data!
I am using your dataset in my research (merged Google and Apple data: https://github.com/ActiveConclusion/COVID19_mobility/blob/master/summary_reports/summary_report_regions.csv), but I don't know how correctly write a reference for it. I don't understand, what does it mean "ActiveConclusion"? Is it some organisation or a simple nickname? Could you elaborate, please, how is better to cite your dataset for publication?
Thank you very much!

Apple report update doesn't work

Recently, Apple changed the report format (added more regions and columns), so currently processing their data doesn't work. I will try to fix this issue in 1-2 days.

Google: updates in some categories are temporarily suspended

From Google Community Mobility Reports website:

2020-09-22 20:45 GMT‎

We have temporarily suspended updates in some categories to prepare for an improvement to how we’ll compute retail, recreation and other categories. The upcoming update will ensure consistency in the way that the data is being reported. We are working to resume updates as soon as possible.

Time series data

Thanks so much for posting this! I'm hoping to track mobility by date... the graphs Google released clearly show trends by county by date. Do you know of any way to scrape or otherwise obtain that information? Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.