aByes is a Python package for Bayesian A/B Testing, which supports two main decision rules:

• Region Of Practical Equivalence (as in the paper Bayesian estimation supersedes the t-test, J. K. Kruschke, Journal of Experimental Psychology, 2012)
• Expected Loss (as discussed in Bayesian A/B Testing at VWO, C. Stucchio)

A lot of the underlying theory is discussed in this blog post.

• In your target folder, clone the repository with the command:

  git clone https://github.com/cbellei/abyes.git

• Then, inside the same folder (as always, it is advisable to use a virtual environment):

  pip install .

• To check that the package has been installed, in the Python shell type:

  import abyes

• If everything works correctly, the package will be imported without errors.

* aByes is tested on Python 3.5 and depends on NumPy, Scipy, Matplotlib, Pymc3 (see requirements.txt for version information).

The main steps to run the analysis of an A/B experiment are:

• Aggregate the data for the "A" and "B" variations in a List of numpy arrays
• Decide how to do the analysis. Options are: 1. analytic solution; 2. MCMC solution (using PyMC3); 3. compare the analytic and MCMC solutions
• Set decision rule. Options are: 1. ROPE method; 2. Expected Loss method
• Set parameter to use for the decision. Options are: 1. Lift (difference in means); 2. Effect size

These and many more examples and instructions can be found in this blogpost.

• In IPython, type:

  import abyes as ab
  import numpy as np
  
  data = [np.random.binomial(1, 0.4, size=10000), np.random.binomial(1, 0.5, size=10000)]
  exp = ab.AbExp(method='analytic', decision_var = 'lift', rule='rope', rope=(-0.01,0.01), plot=True)
  exp.experiment(data)

• This will plot the posterior distribution:

  

• It will then give the following result:

  *** abyes ***
  
  Method = analytic
  Decision Rule = rope
  Alpha = 0.95
  Rope = (-0.01, 0.01)
  Decision Variable = lift
  
  Result is conclusive: B variant is winner!

• There are many more examples available in the file example.py, which can be run from the root directory with the command:

  python abyes/examples/examples.py

Currently, aByes:

• only focuses on conversion rate experiments
• allows for only two variants at a time to be tested

These shortcomings may be improved in future versions of aByes. (Feel free to fork the project and make these improvements yourself!)

Apache License, Version 2.0