A/B testing with a Bayesian approach

Questions
1

Hi,

I have recently been working on an A/B testing problem and have analyzed it the “Frequentist” way.
However, I would like to do it the Bayesian way now and compare. I am not sure if we can can do these kind of questions here so pardon my ignorance. Here’s what I have :

   A total population size of 380000, with a treatment size = 350000, control size = 30000.

The prior conversion rates for both groups were almost similar before applying the treatment to the treatment group --> 0.85 for the control group, 0.86 for the treatment group.

After applying treatment to the treatment group, this is the situation :
Control : Converted = 291, Not-converted = 29709
Target : Converted = 2987, Not-converted = 347013

Can somebody help me with how to go about it and do a bayesian analysis of it to see if the treatment group turned out to be better than the control?

2

Hi Ali,
I think this use-case is perfectly appropriate for the Bayesian framework. There is a good tutorial about that on PyMC’s website, porting Kruschke’s paper to PyMC.

Chapter 10 of McElreath’s Rethinking also has a very good section on treatment effects on chimpanzees – see PyMC3 port of the first edition. The second edition implements it in an even more obvious way, but you’ll have to wait a little for the PyMC port – we are currently working on it.

Hope this helps :vulcan_salute:

1 Like