Best Practice AB Test Difference in Means

Questions
1

I am aware of several blog posts about AB test with hierarchical bayes in terms of comparison of proportions. E.g. https://blog.dominodatalab.com/ab-testing-with-hierarchical-models-in-python/

This includes a prior for a and b of the beta that comes from Gelman.

My question is

  1. if there is a best practice for comparing means (versus proportions)
  2. if the below is a reasonably good approach in Pymc3?
  3. Does a hierarchical approach like this tend to be better than a non-hierarchical approach like taken here: https://joomik.github.io/abtesting/

Given just two groups:

#generate groups
y1=stats.norm.rvs(loc=79,scale=9.639141, size=93)
y2=stats.norm.rvs(loc=81.8064,scale=8.849291, size=93)
y=pd.DataFrame(np.hstack([y1,y2]),columns=["y"])
y['group_indx']=np.repeat((0,1),93)


with pm.Model() as aa_model:
    
    #hyper priors 
    mu_a= pm.Normal('mu_a',y.values.mean() , y.values.std() * 3)
    sigma_a = pm.HalfCauchy('sigma_a',5)
    
    
    #varying intercept prior (creates 2)
    a_group = pm.Normal('a_group',mu=mu_a,sd=sigma_a, shape=2 )
    
    #Other priors for the model (for the T dist likelihood)
    v = pm.Exponential('ν', 1/29.) +1
    sigma_y= pm.HalfCauchy('sigma_y',5)
    
    #what we are interested in (the intercepts)
    y_hat= a_group[y.group_indx.values]  #these are [0,0,0,,,1,1,1....1] relating to the 2 groups
    
    diff_of_means = pm.Deterministic('difference of means 2-1', a_group[1] - a_group[0])
    
    obs = pm.StudentT('observed_values',nu= v, mu=y_hat, sd=sigma_y, observed=y.y.values)
    
    trace_ab = pm.sample(10000, tune=20000)
1 Like
2

My experience is that a hyper-prior usually helps the convergence. So your model looks fine to me - I might use a HalfNormal instead of HalfCauchy tho (although the difference would be really small for model like this)

3

Thank you for confirmation that I am doing this correctly!