A/B testing and Multi-armed Bandits Made Easy

See interactive notebook frequentist.ipynb / Colab / Kaggle Kernel

Frequentist A/B testing is one of the most used and abused statistical methods in the world. This article starts with a simple problem of comparing two online ads campaigns (or teatments, user interfaces or slot machines). It outlines several useful statistical concepts and how we exploit them to solve our problem. At the end, it acknowledges some common pitfalls we face when doing a frequentist A/B test and proposes some possible solutions to a more robust A/B testing. Readers are encouraged to tinker with the widgets provided in order to explore the impacts of each parameter.

See interactive notebook bayesian.ipynb / Kaggle Kernel

We reuse the simple problem of comparing two online ads campaigns (or teatments, user interfaces or slot machines). We details how Bayesian A/B test is conducted and highlights the differences between it and the frequentist approaches. Readers are encouraged to tinker with the widgets provided in order to explore the impacts of each parameter.

See interactive notebook mab.ipynb / Kaggle Kernel

Frequentist and Bayesian A/B tests require you to divide your traffic into arbitrary groups for a period of time, then perform statistical tests based on the results. By definition, this forces us to divert out traffic to suboptimal variations during the test period, resulting in lower overall conversion rates. On the other hand, multi-barmed bandit appraoch (MAB) dynamically adjusts the percentage of traffic shown to each variation according to how they have performed so far during the test, resulting in smaller loss in conversion rates.

To install for the first time:

git clone https://github.com/cstorm125/abtestoo.git
cd abtestoo
pip install . #normal
pip install -e . #dev

To upgrade:

cd abtestoo
git pull

• plotnine
• numpy
• pandas

• N=10^9: Automated Experimentation at Scale
• Big experiments: Big data’s friend for making decisions
• Designing and Deploying Online Field Experiments

• korakot for notebook conversion to run in Colab