Move this document to the wiki, and include a link in the 'Help' section of exercises, with a short paragraph on getting help and avoiding plagiarism when coding.

https://github.com/briatte/dsr/blob/master/dsr-03/02-eu-mood/eu-mood.r

• For teaching purposes, the data cleaning part should appear in a separate script, with full comments, and broken down, not as a 'pipeline' of chained functions.
• Have years on the x-axis?

Look at student feedback -- 'confusing wording' and all.

• include warning about exercise being deliberately more vague than previous ones
• makes all questions MCQs -- easier to grade, less vague!
• clarify scenario (not a replication of the article, just inspired by it)

• All weeks
  • Unify slides ('practice session' slides are different in Weeks 1-10 and in Weeks 11-12)
  • Clean up the readings
    • Focus on as a few core handbooks as possible
    • Cite extra references in the slides/emails
  • Sync the 'readings' slides with the lists of readings + videos sent by email
  • Sync with 'official' syllabus
• Week 3
  • Fix #10
  • Fix #11
  • Lose countrycode dependency?
• Week 4
  • Fix #11 (again?!)
  • Fix #9
• Week 5
• Week 6
  • Refer students to readings, and to this for distributions: https://www.statology.org/tutorials/
  • Link from example NYT article with confidence intervals
• Week 7
  • Slides required!!!
  • Debrief email required!!!
  • Refer students to this tutorial as wrap-up on descriptive stats, correlation and SEs
• Week 8
  • Update Bartels data using recent BLS data?
  • Try to code an EU 2012 membership dummy and add it to Exercise 8?
    • use this: https://jfjelstul.github.io/eums/index.html
    • Current europe dummy, minus Albania, Bosnia, Croatia, Kosovo
  • … or try a Eurozone membership dummy?
    • use this: https://jfjelstul.github.io/eums/index.html
  • Add a mention of Schrodt 2010 in the slides?
  • Cite other extra references, e.g. book on interactions, Breiman's two cultures?
  • Mention the many model tutorials there: https://ourcodingclub.github.io/tutorials.html
  • Mention the many stat/ML tutorials there: https://www.statology.org/machine-learning-tutorials/
• Week 9 (nonlinearity/logit)
  • Slides required!!! (use those from this year's Session 10)
  • Debrief email required!!!
  • Show Jackman graphs of US votes?
• Week 10 (surveys, ex-Week 6)
  • Debrief email required!!!
  • Slides required!!! (cite Gelman 2006)
  • Cover survey data with dummies -- use Chris Hanretty example on same-sex vote?
  • Code Arab Barometer example with survey weights?
  • Code ESS example with weights?
    • In that case, make the exercise use them too?
  • survey exercise using LFS data (relevant to student internships)
• Week 11
  • Debrief email required!!!
• Week 12
  • Document population density fallacies: problem: US map, solution: cartograms (using Plotly?)
    • better maps, esp. US → incl. electoral choropleth from IDA 2013?
    • see comment below on Week 12
  • bonus: SQL + maps/geolocation + R Markdown
    • replace final feedback exercise with this? make it a bonus demo
    • nuclear plants, with SQL: GeoNuclearData-master.zip
    • make it a knittable document: see https://rstudio4edu.github.io/rstudio4edu-book/rmd-templates.html
  • bonus: Web scraping
    • En Marche! 2023 example?
    • Examples from IDA 2013? See #28
    • RSelenium example?
  • Mention the many spatial data tutorials there: https://ourcodingclub.github.io/tutorials.html
  • Debrief email required!!!
    • include link to Spotify playlist
    • include link on APC effects: "America’s Liberal Social Climate and Trends: Change in 283 General Social Survey Variables between and within US Birth Cohorts, 1972–2018" (Hout, POQ, 2021)
    • include link to explanation video on life expectancy (used 4 times as an example variable)
    • include link to https://in-pursuit-of-development.simplecast.com/episodes/indermit-gill
    • include link to Silge and Robinson vidcasts
      • Silge, Topic modeling with R and tidy data principles
    • include link to Savage on Bourdieu
    • how to stay updated on new R stuff
      • useR, rstudio::conf, now posit::conf
      • R Weekly, R Views
      • CRAN Task Views
      • Twitter
    • how to learn moar
      • courses, e.g. SCISS, Coursera, DataCamp
      • read through other courses → link to wiki page
      • find R code on GitHub
      • StackOverflow, CrossValidated

Ideas for more graded exercises

See #27

More (bonus) examples

See #28

Teaching resources

https://education.rstudio.com/teach/

• There are sessions without exercises: Week 6 (association), and Week 11 (classification).
• The exercise for Week 10 (logit) is a follow-up to Week 8 (linear regression), but leave it that way.
• The exercise for Week 12 is a no-code exercise.

1. 'secret weapon' on births and female educ.

• have students start with average over years
• use QOG, or ask for students to assemble the data themselves
• use a t-test instead of linear regression, and make it the exercise for Week 6?

2. regression on survey + PCA on aggregated vars

• Use regional data?
• Make it exercise for Week 11?

3. complex data wrangling with EES 2019 / PartyFacts

• aggregate country/party-level answers
• complex merge of DTA/CSV (CSV has party names)
• add a merge to e.g. PartyFacts?
• compute PTV by party family

… when to have that?

Shown both in dsr-03/01-covid-income and in dsr-04/01-debt. The first one needs to be properly documented, whereas the second one can probably go.

https://luukvdmeer.github.io/sfnetworks/

This one is complex enough to be its own issue…

Weighting guide

https://www.europeansocialsurvey.org/methodology/ess_methodology/data_processing_archiving/weighting.html
https://www.europeansocialsurvey.org/docs/methodology/ESS_weighting_data_1_1.pdf

From the weighting guide, v1.1 (2020), page 7:

From round 9 onwards, all the necessary sample design indicators and weights are already included in the integrated (second release) data file, but if you are working with data from earlier rounds you will first need to merge the sample design indicators on to the main data file. For rounds 7 and 8, the sample design indicators are in the integrated SDDF (sample design data file), so you need to merge this file with the main integrated (questionnaire data) file. For rounds 1 to 6, sample design indicators are stored in a separate file for each country (and files are missing for some countries in some rounds), so you would need to merge several files. Furthermore, for these rounds the indicators psu and stratify have not been recoded in a manner suitable for cross-country analysis, so you will need to do this if you are analysing data from more than one country. Follow the guidance in section 2 of Kaminska & Lynn (2017) and ensure that each value is exclusive to one country.

The guide asks for the creation of anweight ('analytical weights') from the following variables:

# R, data.table syntax
data1[, anweight := pspwght * pweight * 10e3]
# Stata
# gen anweight=pspwght*pweight

Once anweight exists, weighting guide instructs the following design:

# R
svydesign(ids = ~psu, strata = ~stratum, weights = ~anweight, data = data1)
# Stata
# svyset psu [pweight=anweight], strata(stratum)

Details on analytical weights (ESS9+)

Quoting again from the weighting guide:

It is constructed by first deriving the design weight, then applying a post-stratification adjustment, and then a population size adjustment. Further details of how the weights are derived are documented in the round-specific report on the production of weights. Starting from Round 9, anweight is provided for you in the integrated data file. If you are using data from earlier ESS rounds, you can derive anweight yourself.

Full range of weighting variables, quoted from ESS9 codebook:

• idno - Respondent's identification number
• cntry - Country
• dweight - Design weight
• pspwght - Post-stratification weight including design weight
• pweight - Population size weight (must be combined with dweight or pspwght)
• anweight - Analysis weight
• prob - Sampling probability
• stratum - Sampling stratum
• psu - Primary sampling unit

Notes:

• pspwght includes dweight
• anweight is just the product of pspwght and pweight
• no obvious use for prob

Discussions

InductiveStep/R-notes#1
ropensci/essurvey#39
ropensci/essurvey#9 (comment)

Second link right above recommends the following for ESS4:

svydesign(
  ids = ~ psu + idno, # further comment at the link: specifying just `psu` would be enough
  strata = ~ stratify,
  weights = ~ dweight,
  nest = TRUE,
  data = ess4gb
)

Example: Andi Fugard, ESS9

Intermediate Quantitative Social Research, Birkbeck, University of London (2017-2020)
https://inductivestep.github.io/R-notes/complex-surveys.html

Working on a multi-country example:

# using srvyr
as_survey_design(
  ids = idno, # instead of `psu` or `psu + idno` because `psu` is not in ESS9?
  strata = cntry,
  nest = TRUE,
  weights = pspwght
)

From the text:

The nest option takes account of the ids being nested within strata: in other words the same ID is used more than once across the dataset but only once in a country.

Example: Federico Vegetti, ESS7

Introduction to Survey Statistics, University of Heidelberg, 2018
https://federicovegetti.github.io/teaching/heidelberg_2018/lab/sst_lab_day2.html

When working on countries separately:

# using srvyr
as_survey_design(weights = c(dweight, pspwght)) %>%
  group_by(cntry) %>%
  # etc.

# ... doesn't pspwght include dweight?
# ... what about stratum? psu?

When working on all countries together:

# using srvyr
as_survey(weights = c(dweight, pspwght, pweight))

Example: Daniel Oberski, ESS7

http://asdfree.com/european-social-survey-ess.html

Working on a single country (Belgium) after merging the data to the SDDF file:

svydesign(
  ids = ~psu ,
  strata = ~stratify,
  probs = ~prob,
  data = ess_df
)

https://vincentarelbundock.github.io/WDI/ by @vincentarelbundock (current choice)
https://github.com/gshs-ornl/wbstats by @jpiburn (more dependencies)
https://github.com/jpiburn/wbdata also by @jpiburn (unreleased?)

Does one of them also allow to download WB shapefiles?

https://iqss.github.io/dss-workshops/

… mention in first session/slides?

Had forgotten about this…
https://www.r-bloggers.com/2014/12/cartography-with-complex-survey-data/

And especially:
https://www.r-bloggers.com/2014/12/maps-and-the-art-of-survey-weighted-maintenance/

Perhaps the code can be adapted to run again? Would have to be quick enough to be run in class, though.

ESS

Uses kriging with fields.

https://github.com/davidbrae/swmap/blob/master/how%20to%20map%20the%20european%20social%20survey.R

DHS

Uses KDE with prevR, by Joseph: see the intro vignette.

http://asdfree.com/demographic-and-health-surveys-dhs.html
https://github.com/davidbrae/swmap/blob/master/how%20to%20map%20the%20demographic%20and%20health%20surveys.R

• https://github.com/SciutoAlex/Historical-Travel-of-US-Secretary-State
  • rewrite scraper?
  • use data for mapping examples

https://strengejacke.github.io/regressionmodels/

• https://github.com/mdsumner/spbabel
• https://github.com/mdsumner/spdplyr
• https://github.com/edzer/sfr
• https://cran.r-project.org/web/packages/tmap/index.html
• https://github.com/Groupe-ElementR/cartography

… and the list goes on: see my blog post on the issue (kind of).

Some folks are putting ideas together on the topic.

U.S. spatial data: https://github.com/martinjhnhadley/statesRcontiguous

rlist, by @renkun-ken, looks like a brilliant way to manipulate lists. My only regret is that the functions are named list.verb instead of list_verb.

Suggestion – Use it if lapply usage becomes cryptic.

pipeR looks cool too, even though I would prefer to keep things simple and stick with just %>%.

For an advanced version of the course, some notes on the stuff that were excluded due to time constraints:

1. Data manipulation (would deserve an extra session)

• Factors, more in depth
• More merging and pivoting
• Row-wise operations
• New dplyr 1.1.0 functions

2. Surveys

• Labelled data (treated in passing; related to the point above re: factors)
• Survey weights

3. Databases: SQL etc. (with passing mention of big data)
4. APIs and Web scraping
5. Models -- Logit beyond logit: ordered logit, multinomial logit
6. Models -- panel data, FE/RE, standard error corrections
7. Models -- mixed models
8. CVS -- Git/GitHub
9. Dynamic documents -- R Markdown, Quarto
10. Networks (data, viz, models)
11. Text (this time covering e.g. topic models)
12. JavaScript visualization libraries

So, basically, this advanced course could be, over 12 sessions:

• 2 'refresher' sessions to get started again
  • introduce Git/GitHub
  • introduce R Markdown
• 4 more sessions on data
• 1 'refresher' session on (linear) models
• 3 more sessions on models (with a refresher on logit in the first one)
• 1 'extra' session on text
• 1 'extra' session on networks

(Still no space for JS libraries, fair enough.)

This neglects PCA-style stuff and ML, which should probably be its own project, centered on tidymodels, covering random forests, gradient-boosted trees, etc.

dsr-03/01-covid-income needs more structure, and some steps need to be fully documented, e.g.

• better document the code
  • locale()
  • as.Date()
  • skip in read_excel
• merging
  • explain merging in detail (use code and comments from dsr-04/01-debt)

Also:

• The header of the script comes from QUANTI2 and should be changed.
• The "quick check" part should show str, head and tail, View and glimpse.
• hard-code OECD membership: document source in README
  • remove dependency to memberstates::memberstates$oecd$iso3c
• add ggsave at end

Slides

Current size of slide sets (cap at 25, except Week 1). Revise readings, practice sessions and exercises, and include screenshots of videos when relevant.

• 1. 37 -- OK, cap at ~ 40
• 2. 24 -- OK
• 3. 33 -- slightly too long, cut down a bit
• 4. 25 -- OK
• 5. 17 -- expand (description, sampling) -- add 'how to get help online' for Exercise 5
• 6. 20 -- expand a bit (association) -- cover bootstrapping, Bayesian reasoning?
• 7. TODO (correlation) -- cover bivariate OLS geometry
• 8. 15 -- expand (regression)
  • expand further with slides specifically on regression output (use SRQM)
• 9. 16 -- expand (logit) -- cover nonlinearity via LOESS, splines?
• 10. TODO (surveys) -- cover data, again, data wrangling, again, weighting
• 11. 30 -- slightly too long, cut down a bit (classification)
• 12. 11 -- expand a bit

Syllabus

• Port essentials from the current syllabus from PDF to Google Docs Done.
• Continue moving session recaps from syllabus to DSR-outline-2.txt to GitHub README and emails
• Fix #28
• Possibly use stuff from EMSS-emails-2013
  • Pandoc template
  • ‘Bilan’ section
  • Examples
    • https://github.com/briatte/aad/tree/master/ess4-israel
    • ESS5 France
    • Reinhart and Rogoff
    • Piketty and Saez
• Look at more old stuff esp.
  • QUANTI1-2020
  • QUANTI2-2019
• Store the emails online: GitHub wiki? emails/ folder?

Other courses and tutorials

Make better use of great tutorials:

• https://inductivestep.github.io/R-notes/
• Vegetti, surveys
• https://www.statology.org/tutorials/ (stats)
• https://www.statology.org/r-guides/ (R)
• https://ourcodingclub.github.io/tutorials.html
• UCLA: https://stats.oarc.ucla.edu/r/
• https://m-clark.github.io/documents.html (more advanced models)
• Tidy models: https://www.tmwr.org/ (actually a book, recommend in Session 11?)
• David Robinson's videos
• Julia Silge's videos

Finish digging into (and reorganising…) those:

• https://r-bootcamp.netlify.app/ (biologists)
• https://ready4r.netlify.app/labbook/ (biologists)
• http://post8000.svmiller.com/lab-scripts/ (poli-sci)
  • http://post8000.svmiller.com/lab-scripts/logistic-regression-lab.html (logit)
• https://grodri.github.io/glms/ (and his other demography courses perhaps)
• https://chester.rbind.io/teaching/
• https://rudeboybert.rbind.io/
• https://rudeboybert.github.io/MATH116/problem-sets/ (math/stat)
• https://www.amelia.mn/STAT220labs
• QUANTI1-courses folder
• QUANTI1-handbooks folder
• QUANTI2-courses folder
• QUANTI2-handbooks folder
• DSR-handbooks folder
• DSR-courses folder
• Any other 'courses' folder…

Possible additions for the wiki:

• https://github.com/UChicago-pol-methods/
• https://github.com/UChicago-pol-methods/IntroQSS-F22
• https://github.com/UChicago-pol-methods/plsc-40502-statistical-models
• https://github.com/UChicago-pol-methods/plsc-30600-causal-inference

Paper to turn into an exercise

Amelia McNamara, Nick Horton, "Wrangling categorical data in R," citing her website:

Wrangling categorical data in R, a paper co-authored with Nick Horton. This paper describes some common mistakes data analysts make when working with categorical data (factors) in R. The paper was published jointly in The American Statistician, Vol. 72, Issue 1 and as a pre-print in the Practical Data Science for Stats collection on PeerJ.

• Final exercise is great. Use it.
• Cite in readings.

Readings

Finalize the list:

• Finalize
  • Sync emails with list (working backwards…)
• Document readings in slides, with "(on Google Drive)" markers when relevant
• List (almost) all material mentioned in emails and slides
• Establish session-per-session roadmap
• Copy to syllabus Link to wiki in syllabus

Handbooks:

• Make a choice between Irizarry and Rodrigues, (1) assigning both is too much, (2) too much overlap, (3) need space to assign more from Wickham and Grolemund
• Add Gelman et al. chapters
• Add Fogarty - Quantitative Social Science Data with R. An Introduction (2019) (videos and screencasts)
  • (Week 1, ‘Introduction to R and RStudio’) videos, screencasts
  • (Week 3?) Finding Data, Finding Data
  • (Week 3) Data Management, Data Management
  • (Week 3, again) Variables & Manipulation, Variables & Manipulation
  • (Week 4) Visualising Data, Visualising Data
  • (Week 5) Univariate & Descriptive Statistics, Univariate & Descriptive Statistics
  • (Week 6, ‘Hypothesis Testing’) videos, screencasts
  • (Week 7) Bivariate Analysis, Bivariate Analysis
  • (Week 8, 7 perhaps if easy) linear Regression & Model Building, Linear Regression & Model Building
  • (Week 8) OLS Assumptions & Diagnostic Testing, OLS Assumptions & Diagnostic Testing
  • (Week 12, wrap-up) https://study.sagepub.com/fogarty/student-resources/r-screencast-tutorials/chapter-13-putting-it-all-together
• Add Baumer et al. -- https://mdsr-book.github.io/mdsr2e/
• (Re)add Peng? (probably not) (nope, use Irizarry instead)
• Add 'ModernDive' book, Statistical Inference via Data Science to main readings
  • Session 1: https://moderndive.com/1-getting-started.html
  • Session 4: https://moderndive.com/2-viz.html
  • Session 3: https://moderndive.com/3-wrangling.html
  • Session 5 (glossary): https://moderndive.com/A-appendixA.html
  • Session 5 (concepts, illustrated): https://moderndive.com/B-appendixB.html
  • Session 5 (perhaps): https://moderndive.com/7-sampling.html
  • Session 5(also, also perhaps): https://moderndive.com/8-confidence-intervals.html
  • Session 6: https://moderndive.com/9-hypothesis-testing.html
  • Session 8 (or 7 perhaps): https://moderndive.com/5-regression.html
  • Session 8, again: https://moderndive.com/6-multiple-regression.html
  • Session 8, again, perhaps: https://moderndive.com/10-inference-for-regression.html
• Add stuff from syllabus to slides or readings/roadmap
  • Add 'Quanti. Soc. Sci. Data' (3)
  • Add R Graphics Cookbook (4)
  • Add CRAN Task Views (12)
  • Add StackOverflow (and CrossValidated) (5, after instructions for first exercise)
• Document optional scientific articles (the readings are numbered in ref. to the sessions, but listing them all in the syllabus and/or the wiki should be enough)
  • (1) Amelia McNamara, Key attributes of a modern statistical computing tool: The American Statistician, Vol. 73, Issue 4, preprint
  • (1) McCullough, B.D. and Yalta, A.T. 2013. “Spreadsheets in the Cloud – Not Ready Yet,” Journal of Statistical Software 52(7). doi:10.18637/jss.v052.i07
  • (3) Broman, K.W. and Woo, K.H. 2018. “Data Organization in Spreadsheets,” The American Statistician 78: 2–10.
    doi:10.1080/00031305.2017.1375989 -- https://kbroman.org/dataorg/
  • (3) Ellis, S.E. and Leek, J.T. 2017. "How to Share Data for Collaboration," PeerJ Preprints 5:e3139v5.
    doi:10.7287/peerj.preprints.3139v5
  • (3) Wickham, Tidy Data, Journal of Statistical Software, 2014
  • (4) Wickham, A Layered Grammar of Graphics, Journal of Computational and Graphical Statistics, 2010
  • (4?8?) Wickham et al, Visualizing Statistical Models.
    Published in Statistical Analysis and Data Mining.
  • (8) Schrodt, P.A. 2010. “Seven Deadly Sins of Contemporary Quantitative Political Analysis.” APSA 2010 Annual Meeting. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1661045
  • (8) Kam, C. and Franzese, R. 2007. Modeling and Interpreting Interactive Hypotheses in Regression Analysis. University of Michigan Press, ch. 2 (‘Interactions in social science’) and 3 (‘Theory to practice’). https://www.press.umich.edu//206920
  • (10) Damico, surveys: https://journal.r-project.org/archive/2009/RJ-2009-018/index.html
  • (10) Gelman, A. 2007. “Struggles with Survey Weighting and Regression Modeling.” Statistical Science. http://www.stat.columbia.edu/~gelman/research/published/STS226.pdf
  • (11) Breiman, L. 2001. “Statistical Modeling: The Two Cultures.” Statistical Science. https://doi.org/10.1214/ss/1009213726

… with survey and srvyr

Also show

• downloading data and docs
• reading fixed-width files

https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Dataset_Documentation/NHIS/

Code:

https://github.com/mlaviolet/Trends/blob/master/NHIS/nhis.R
http://asdfree.com/national-health-interview-survey-nhis.html

Working backwards…

QUANTI1 (also add handbooks):

1. Software [ex: cty-level CSV]
2. (2+3) Data + Visualization [csv via ex, CWS/xlsx, QOG-cs/dta, ex: cty]
3. (4+5) Description + sampling (distributions) [svy, ex: svy]
4. (6+7) Association + surveys (labels, factors) [svy, EXAM: svy]
5. (8+9) Correlation + OLS.1 (simple, residuals) [cty, ex: cty/covid]
6. (XX+10) OLS.2 (dummies, interactions) + Diagnostics & mfx [svy, EXAM: svy]

QUANTI2:

• Whatever's not reused from data wrangling goes to Session 4
• Whatever's not reused from data viz. goes to Session 5
• Session on classification goes to Session 12 (if ever taught again)

TODO

• Actually do the DSR → QUANTI move
• Create the two repos

In Sessions 6, 7 and 8, possibly 9?

https://easystats.github.io/parameters
https://easystats.github.io/parameters/articles/standardize_parameters_effsize.html

… to replace protein consumption in Session 11?
… to combine it with correlation analysis?

https://drsimonj.svbtle.com/how-to-create-correlation-network-plots-with-corrr-and-ggraph
https://fivethirtyeight.com/features/dear-mona-followup-where-do-people-drink-the-most-beer-wine-and-spirits/

Hello their,

I tried running this code to pull the most recenct CPJ data into, R, but the link seemingly does not work and I can't find the CPJ database that goes up to 2017 in Excel. Would you happen to possess that dataset to update the link and/or code?

u = "https://www.cpj.org/killed/cpj-database.xls"
f = basename(u) # New_York_Times_Brexit_coverage.xlsx

if (!file.exists(f))
  download.file(u, f, mode = "wb")

Thanks for taking the time to look at this pull-request.

http://asdfree.com/american-national-election-study-anes.html
http://asdfree.com/european-social-survey-ess.html
http://asdfree.com/general-social-survey-gss.html
http://asdfree.com/survey-of-health-ageing-and-retirement-in-europe-share.html
http://asdfree.com/world-values-survey-wvs.html

Also add

• Arab Barometer?
• Afro Barometer?

http://faculty.missouri.edu/williamslaro/mipdata.html

Email from Laron K. Williams to EPSA mailing-list below.

Hi everyone.

It is my pleasure to announce the release of the 'Most Important Problem' Dataset (MIPD). It is the result of a collaborative project on Americans' issue attentiveness with two Ph.D. students at the University of Missouri (Colton Heffington and Brandon Beomseob Park).

The MIPD collects all available individual-level responses to the 'most important problem facing this country' question in America from 1939 to June 2015. In addition to providing a sense of how individuals prioritize problems, the MIPD paints a picture of Americans' evolving issue attention over time. There are two datasets connected to the MIPD:

a) MIPD: This dataset contains individual-level responses (from over 670 surveys totaling almost a million respondents) to the MIP question, in addition to demographics, partisan preferences (including vote intention, previous vote, ideology and party identification), approval (general approval and approval of specific policies), economic evaluations (both retrospective and prospective, personal and national), and party competency (best party to address MIP, best party to address specific problems). We code the MIP responses into three coding schemes to ease the incorporation of other datasets: manifesto research group, the comparative agendas project, and Matt Singer's (2011) coding scheme. In addition to being the largest possible data collection of individual-level responses dealing with issue attention, we could certainly envision other uses of the dataset ranging from studies of vote intention, party competency, and partisan rationalization.

b) MIPD Aggregate: This dataset contains the aggregate percentage (weighted by population weights, if available) of respondents identifying that problem as the MIP. These aggregate data are available monthly, quarterly, and annually. This dataset would be a valuable component in an overall portrait of dynamic representation, and in observing how public opinion shifts in response to changing domestic and foreign circumstances.

The data are available from my website (http://faculty.missouri.edu/williamslaro/mipdata.html ) or from my Dataverse (https://dataverse.harvard.edu/dataverse/laronwilliams ).

If you use it, please cite: Colton Heffington, Brandon Beomseob Park and Laron K. Williams (forthcoming). "The 'Most Important Problem' Dataset (MIPD): A New Dataset on American Issue Importance", Conflict Management and Peace Science.

Thanks in advance. Feel free to contact me if you have any questions!

Laron

• #33
• Add LFS (mentioned by a student) (see below)
• Add NHIS? (see below)
• Add Arab Barometer, with weights this time?
• Add Afro Barometer?

Other surveys of interest

• ANES: http://asdfree.com/american-national-election-study-anes.html
  • https://tidy-survey-r.github.io/tidy-survey-short-course/
  • https://github.com/tidy-survey-r/tidy-survey-book
• GSS: http://asdfree.com/general-social-survey-gss.html
• SHARE: http://asdfree.com/survey-of-health-ageing-and-retirement-in-europe-share.html
• WVS: http://asdfree.com/world-values-survey-wvs.html

Tidy survey book

https://github.com/tidy-survey-r/tidy-survey-book
https://tidy-survey-r.github.io/tidy-survey-book/

• Assign chapter(s) when done?
• Use ANES code / examples?

LFS

https://ec.europa.eu/eurostat/web/lfs
https://ec.europa.eu/eurostat/web/microdata/european-union-labour-force-survey
https://www.gesis.org/en/missy/metadata/EU-LFS/
https://ec.europa.eu/eurostat/web/microdata/public-microdata/labour-force-survey

ILO ones: https://www.ilo.org/surveyLib/index.php/collections/LFS

NHIS

Code:

https://github.com/mlaviolet/Trends/blob/master/NHIS/nhis.R
http://asdfree.com/national-health-interview-survey-nhis.html

Also show:

• downloading data and docs
• reading fixed-width files

https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Dataset_Documentation/NHIS/

• dsr-03/01-covid-income
  • better document the code
  • explain merging in detail (use code and comments from dsr-04/01-debt)
  • hard-code OECD membership (document source in README)
  • add ggsave at end
• dsr-03/02-eu-mood
  • lose countrycode dependency?
• dsr-04/01-debt
  • remove merge
  • adjust README accordingly
  • focus on building line plots, ending with small multiples
• dsr-04/03-anscombe
  • cover base plots
  • cover lattice?

… as 'bonus' demos?

https://github.com/briatte/ida

• Not all examples are very academic, some are just fun data
• It would be nice to turn the course into a website like I did back then, but it would be too much work to write up everything.
• Another solution would be to just update the code from IDA and republish that…

Candidates for Week 2 (functions)

• Writing up a BMI function
• Possible conflicts between roommates

Candidates for Week 3 (data)

• Reading Obama 2008 v 2012 from Google Sheets
• Factors, using WDI data
  • Not the best example, but having a bonus on factors would be helpful
  • Same goes for dates
• Reshaping and plotting data: Case-Shiller Index, and US homicides
  • Could also fit in Week 4, but might be better to illustrate pivots and aggregation
• UK polling accuracy, 1984-2016
  • Data stop in 2016 (and includes general election results!)
  • Still a good example of importing from Google Sheets + some wrangling

Candidates for Week 4 (plots)

• Ideological polarization in the US Congress
  • There's new data for that
• Plotting an ECDF -- Week 4?
• Building a US recessions plot, bit by bit -- Week 4?
  • Would require finding the original data again

Candidates for Week 8 (linear models)

• Predicting 100m sprint times
  • Data source is gone, but shouldn't be too hard to replace

Candidates for Week 12 (extra stuff)

Maps (covered by #34)

• Isarithmic maps!
  • The code never got posted, but there are instructions
  • Search GitHub for akima map examples?

Web scraping:

• Web scraping a Twitter account (Beijing air pollution), plus minimal time series stuff
• Web scraping FIDE chess rankings
• Web scraping + Herfindhal-Hirschman Index -- Week 12?
  • Data source still looks processable
  • Have it as a Web scraping + R Markdown example?

Text:

• Julian Assange speech(es?)

Candidates for unassigned weeks

Networks:

• My own stuff…

Time series:

• Smoothing Simpsons TV rankings + changepoints
• Detrending the Piketty/Saez top income data

Reproducible/open science:

• That rant deserved to exist in 2013, and still does in 2023…

Shapefiles

• World Bank
• EuroGeographics
• GADM data -- example
• Other world map essentials?

Geolocalisations

• OSM -- example

Data

• ARVIG (Right-Wing Violence in Germany)
• Nuclear Power Database
• US Secretary of State Travel (rewrite Web scraper?)
• Contiguous United States of America

Old links

Everything below relates to discussions and experiments that happened prior to sf.

• https://github.com/mdsumner/spbabel (superseded by sfheaders and/or silicate, as per the author)
• https://github.com/mdsumner/spdplyr (old experiment, archived, as per the author)
• https://github.com/edzer/sfr (now sf)
• tmap (superseded by other options)
• cartography (superseded by mapsf)

Related to an old blog post and to this old issue: #1

For the QUANTI students who ask for it. Make it a section…

• of the syllabus?
• of the 'heuristics' document?
• of the wiki?

https://juba.github.io/tidyverse/
https://larmarange.github.io/analyse-R/
https://larmarange.github.io/guide-R/

Videos:

https://larmarange.github.io/webin-R/
https://larmarange.github.io/webin-R/seances.html

Workload of the course (min. 2, max. 6, recommended: 4):

• 2 hours in class
• 1-2 hour going through the weekly emails and a selection of the listed readings and videos
• 1-2 hours going through the exercise (which will include going back to the readings, and Web searches)
  • should be done in groups

• Choose between moments and e1071
  • Only mention e1071 and its alternative measures?
• Session 9
  • Use ggeffects or marginaleffects -- or both? use in more than one session?
• Session 11
  • Replace car with anything that plots scatterplot matrices?
  • Remove corrr entirely?
  • Remove ggcorrplot entirely? (or find a package that does the same, plus scatterplot matrices?)
  • Keep factoextra and ggfortify
  • Find a way to re-use plotly? e.g. https://plotly-r.com/d-charts.html -- or remove entirely
• Session 12
  • Install sf earlier and have a map in Session 3
  • Replace life expectancy example from Session 12 with more complex example?
  • Add a full-fledged network example to justify installing igraph and ggraph?
  • Add a full-fledged dynamic document example using modelsummary (and possibly texreg)?
  • Make pdftools optional (only recommend it in the slides?)
• Extras
  • Provide an additional example using rvest?
  • Keep WDI, but use it more than once? -- see #21
    • Modify exercise-04 to use WDI but save the data
    • Modify life-expectancy to use WDI and save the data

Checks:

• Check that countrycode is used in multiple sessions
• Check that broom is used in multiple sessions
• Check that ggmosaic is used in multiple sessions
• Check that ggrepel is used in multiple sessions, "", "", "",
• Check that performance is used in multiple sessions
• Check that texreg is used in multiple sessions

https://github.com/briatte/dsr/blob/master/dsr-04/01-debt/debt.r

Right now, it's an example of data wrangling from QUANTI2-2023.

• Remove the RR merge.
  • adjust README accordingly
• Remove the EU accession merge.
• Focus on building line plots, ending with small multiples
• Reintroduce some colouring of the time series

For the session on correlation:

https://ggforce.data-imaginist.com/reference/geom_autopoint.html

https://twitter.com/alexandreafonso/status/1658451225174753280

https://europeangovernanceandpolitics.eui.eu/eui-yougov-solidarity-in-europe-project/
https://cadmus.eui.eu/handle/1814/72778

d <- "2020 SiE year dataset/2020 SiE dataset_dataset.csv" %>%
  readr::read_csv()

count(d, qcountry)
cty <- c("United Kingdom" ,"Denmark" ,"Finland" ,"France" ,"Germany" ,"Sweden" ,"Greece" ,"Hungary" ,"Italy" ,"Lithuania" ,"Netherlands" ,"Poland" ,"Romania" ,"Spain")
d$qcountry <- factor(d$qcountry, labels = cty)

summary(d$weight)

str_subset(names(d), "q20")
d <- mutate_at(d, vars(starts_with("q20")), ~ case_when(.x == 1 ~ 1L, .x == 2 ~ 0L, .x == 3 ~ NA_integer_))

# Austria
count(d, qcountry, q20_1)
tapply(d$q20_1, d$qcountry, mean, na.rm = TRUE)

# TODO: survey weights
# TODO: make dyadic version

Possibly present as a ‘freestyle’ exercise -- here's the data, do whatever you like with it. In this case, provide the data as a truly dyadic dataset, with q20_x merged into a single item, and additional columns corresponding to the model in the paper.

https://github.com/davben/arvig

Not currently covered, might be interesting by using https://vincentarelbundock.github.io/marginaleffects/

Do what the README of exercise-04-answer (not in the repo) recommends doing: below the actual answer (Step 2 of the preston-curve-answer.r script), show a detailed breakdown of the steps.

https://visionscarto.net/global-nuclear-power-database

Some of them probably breach idiotic copyrights, and the links in the README files should do. Not including the docs is in fat a good incentive to go and read those READMEs in the first place.

Possible reorganisation:

• Make it Exercise 1, focusing
  • setting the working directory correctly
  • having installed the tidyverse
  • understanding the stats (optional)
• Make the generative art Example 1

Might be too complex, and having in Session 3 makes sense if the code is updated to show multiple graph engines (base, lattice, ggplot2).