Examples for web-scraping and text analysis with R

Run the following lines first in R console to install required packages:

for (pkg in c("rvest","httr","dplyr","stringr","XML","RCurl","ggplot2","reshape","tm","ggmap")){
 if (!pkg %in% rownames(installed.packages())){install.packages(pkg)}
}

Then, in RStudio,

1. Select File -> New Project... -> Version Control -> Git
2. Paste https://github.com/ubcecon/web-scraping-with-R (the URL for this repo) into the space for Repository URL. Press on Create Project.

or simply clone this repo using your favourite Git client to checkout this repository.

Once the repo is checked out, try replicating the following examples by yourself by opening the corresponding .Rmd files:

Basic principles of web scraping by URL patterns and HTML parsers.

Web scraping for data that are chaging real-time.

HTML/CSS analysis using SelectorGadget and developer tools for rvest & principles of basic text analysis with beautiful wordclouds.

Web scraping from query-based webpages and geocoding.

Relevant R packages and developer tools:

• rvest (HTML parsing): https://github.com/hadley/rvest
• tm (text mining and analysis): http://tm.r-forge.r-project.org/
• SelectorGadget (HTML/CSS analysis): https://selectorgadget.com/

Useful R packages for data cleaning:

• PSID: https://github.com/floswald/psidR
• The World Wealth and Income Database: https://github.com/WIDworld/wid-r-tool
• The Survey of Professional Forecasters: https://github.com/joergrieger/Survey