This course introduces students to experimentation in data science. Particular attention is paid to the formation of causal questions, and the design and analysis of experiments to provide answers to these questions. This topic has increased considerably in importance since 1995, as researchers have learned to think creatively about how to generate data in more scientific ways, and developments in information technology has facilitated the development of better data gathering.

This course begins with a discussion of the issues with causal inference based on observational data. We recognize that many of the decisions that we care about, whether they be business related or theoretically motivated, are essentially causal in nature.

The center of the course builds out an understanding of the mechanics of estimating a causal quantity. We present two major inferential paradigms, one new and one you are likely familiar with. We first present randomization inference as a unifying, intuitive inferential paradigm. We then demonstrate how this paradigm sits in complement to the classical frequentist inferential paradigm. These concepts in hand, we turn focus to the design of experiments and place particular focus both answering the question that we set out to answer, and achieving maximally powered experiments through design.

The tail of the course pursues two parallel tracks. In the first, students form a research question that requires a causal answer and design and implement the experiment that best answers this question. At the same time, new content presented in the course focuses on the practical stumbling blocks in running an experiment and the tests to detect these stumbling blocks.

We hope that each student who completes the course will:

• Become skeptical about claims of causality. When faced with a piece of research on observational data, you should be able to tell stories that illustrate possible flaws in the conclusions.
• Understand why experimentation (generating one’s own data by doing deliberate interventions) solves the basic causal-inference problem. You should be able to describe several examples of successful experiments and what makes you feel confident about their results.
• Appreciate the difference between laboratory experiments and field experiments.
• Appreciate how information systems and websites can be designed to make experimentation easy in the modern online
• Understand how to quantify uncertainty, using confidence intervals and statistical power calculations.
• Understand why control groups and placebos are both important.
• Design, implement, and analyze your own field experiment.
• Appreciate a few examples of what can go wrong in experiments. Examples include administrative glitches that undo random assignment, inability to fully control the treatment (and failure to take this inability into account), and spillovers between subjects.

Computing is conducted primarily in R.

If you are looking to work on something over the break, between semesters, we recommend this course on `data.table`, created by the package author Matthew Dowle, and available for free at DataCamp. I want to acknowledge the inappropriate behavior on the part of the datacamp CEO and their subsequent response. I do not condone this behavior. There cannot be this kind of behavior in our community; not on the part of harassers, nor on the part of institutions that fail to take seriously these occurrences. This Buzzfeed News article provides an accounting of the facts. I want to note that this behavior is not associated with the people who wrote `data.table`, and that if you were to do work through this three-hours long set of lectures, you will not be giving DataCamp any money.

This semester we’re using the UCB Datahub as our compute backbone. You can get to it here. This is a minimal instance – you’re capped at 1GB memory, but it is a really nice way to work on async coding without having to start any machinery of your own. You should be able to knit, save, and edit as you like. The courses’ upstream repository is entirely segmented from your copy of this, so feel free to make any changes that you want. Note, however, that this also means that changes you make in the datahub will not be present on your own fork of the repository. In other words, things that happen in the datahub, stay in the data hub.

We use two books in this course, and read a third book in the second week. We recommend that you buy a paper copy of the two textbooks (we’ve chosen textbooks that have a fair price), and would understand if you digitally read the third book.

• Field Experiments: Design and Analysis is the core textbook for the course. It is available on Amazon for $40 [link].
• Mastering Metrics is the secondary textbook for the course. It is available at Amazon for $20 [link].
• More than Good Intentions is the third book for the course. It is available at Amazon for $10, new, or $3 used [link]. But, you could also read this digitally.

• We have made all the articles we read in the couse available in the repository. However, it is a great practice to get used to establishing a VPN to gain access to all the journal articles that are available through the library subscription service. Instructions for connecting are available on the UCB library website. Journal access is one of the greatest benefits to belonging to a University, we suggest you use it.
• David has made a great resource that has suggestions for further reading. You can access it in this living google doc.

• In weeks where we have problem sets due, we will hold extra office hours on the Friday before the weekend. As well, when you are working through your project design, the instructors will schedule individual one-on-one conversations as necessary with student groups.
• On Saturdays after we turn in problem sets, we will hold extra office hours to review the work that you’ve done and the feedback that you’ve received. For obvious reasons, you can only attend these Saturday OH if you have submitted your via PR.

• Problem Sets (50%, 10% each) A series of problem sets, mostly drawn from FE, many requiring programming or analysis in R.
  • We encourage you to work together on problem sets, because great learning can come out of helping each other get unstuck. We ask that each person independently prepare his or her own problem-set writeup, to demonstrate that you have thought through the ideas and calculations and can explain them on your own. This includes making sure you run any code yourself and can explain how it works. Collaboration is encouraged, but mere copying will be treated as academic dishonesty.
  • At this point, the course has lived for a number of semesters, and we have shared solution sets each semester. We note in particular that struggling with the problems is a key part of the learning in this course. Copying from past solutions constitutes academic dishonesty and will be punished as such; you should know that we have included language in the solutions that will make it clear when something has been merely copied rather than understood.
• Essays (20%, 10% each) You will write two essays in the course. For each essay, you will first complete a round of peer-evaluation and will then submit a final, revised version of your essay for review by the instructor. These peer reviews will not be graded, but instead will be marked for credit/no-credit.
• Class Experiment (25%) In teams of 3-5 students, carry out an experiment that measures a causal effect of interest. See the `./finalProject/` folder for much more information
• Experiment Pilot Data (5%) Pilot data analysis of distribution of outcome variable and covariate balance check
• Late Policy: You’re busy and things come up – kids get sick, parents stop by unannounced, managers ask you to reformat your TPS reports, you learn that your 261 project has accumulated $50,000 in compute costs – we get it. You’ve got five (5) days to turn things in late without penalty, without explanation, and without notice. We’ll count at the end of the semester. After you use those 5, each additional day (or part thereof) comes at the cost of 10% on the assignment. That is, 1% off your end-of-semester total grade. Here’s the other twist though – we need to provide solutions back to your classmates who have completed their work. So, no individual assignment can come in more than 5 days late; any assignment that does will score a zero. If you see ahead of time that you’re going to have a conflict – a major release, a vacation, etc. – talk with your instructor to work out an alternative. We’ll work with you, but the more notice, the better.