# Best Practice AB Test Difference in Means

I am aware of several blog posts about AB test with hierarchical bayes in terms of comparison of proportions. E.g. https://blog.dominodatalab.com/ab-testing-with-hierarchical-models-in-python/

This includes a prior for a and b of the beta that comes from Gelman.

My question is

1. if there is a best practice for comparing means (versus proportions)
2. if the below is a reasonably good approach in Pymc3?
3. Does a hierarchical approach like this tend to be better than a non-hierarchical approach like taken here: https://joomik.github.io/abtesting/

Given just two groups:

``````#generate groups
y1=stats.norm.rvs(loc=79,scale=9.639141, size=93)
y2=stats.norm.rvs(loc=81.8064,scale=8.849291, size=93)
y=pd.DataFrame(np.hstack([y1,y2]),columns=["y"])
y['group_indx']=np.repeat((0,1),93)

with pm.Model() as aa_model:

#hyper priors
mu_a= pm.Normal('mu_a',y.values.mean() , y.values.std() * 3)
sigma_a = pm.HalfCauchy('sigma_a',5)

#varying intercept prior (creates 2)
a_group = pm.Normal('a_group',mu=mu_a,sd=sigma_a, shape=2 )

#Other priors for the model (for the T dist likelihood)
v = pm.Exponential('ν', 1/29.) +1
sigma_y= pm.HalfCauchy('sigma_y',5)

#what we are interested in (the intercepts)
y_hat= a_group[y.group_indx.values]  #these are [0,0,0,,,1,1,1....1] relating to the 2 groups

diff_of_means = pm.Deterministic('difference of means 2-1', a_group - a_group)

obs = pm.StudentT('observed_values',nu= v, mu=y_hat, sd=sigma_y, observed=y.y.values)

trace_ab = pm.sample(10000, tune=20000)
``````

My experience is that a hyper-prior usually helps the convergence. So your model looks fine to me - I might use a HalfNormal instead of HalfCauchy tho (although the difference would be really small for model like this)

Thank you for confirmation that I am doing this correctly!