Bayesian AB Test Duration and Sample Size

Noob Question: I have recently gotten into Bayesian AB testing, but I am lost on how to specify duration of test and number of observation we are looking to observe. Compared to frequentist approaches that have a clear formula for this, how do we a approach this as bayesians?

Maybe a precision-based power analysis could help? You can check Kruschke’s Doing Bayesian Data Analysis chapter 13.

The idea is to iteratively run the model with simulated data until reaching a desired precision (pre-specified HDI width) for one or more parameters in the model. This could provide an estimate of an ideal sample size.

1 Like

Ill check this out! Thank you!

This was perfect. I really like the concept of specifying a desired HDI width and then simulating how many samples you would need to achieve it.

A very practical approach. Thank you again!

1 Like