Hierarchical AB Testing Posterior Delta

Questions
1

Hi
I’m working on a partially pooled AB testing model in the same way that was described in this forum.

Albeit not understanding it fully the problem with the hyperpriors from the post is solved using this source:

import theano.tensor as T
delta=[]
    with pm.Model() as bb_model:
        def ab_likelihood(value):
            a = value[0]
            b = value[1]
            return T.switch(T.or_(T.le(a, 0), T.le(b, 0)),
                            -np.inf,
                            np.log(np.power((a + b), -2.5)))
        ab = pm.DensityDist('ab', ab_likelihood, shape=2, testval=[1.0, 1.0])

        a = ab[0]
        b = ab[1]

        rates = pm.Beta('rates', a, b, shape=7)
        trials = np.array([7543,7214,7375,7539,7356,7426,7336])
        successes = np.array([67,83,70,81,58,75,77])

        obs = pm.Binomial('observed_values', n=trials, p=rates, observed=successes)
        
        delta = pm.Deterministic('deltaAB',obs - obs[0])

    with bb_model:
        bb_trace = pm.sample(10000, tune=5000,random_seed=777)

However the diagnostic for my model do not look as expected and it also takes around 40 mins. There seems to be a problem with the calculation of the “delta” between the distributions.

pm.summary(bb_trace)

summary
summary1291×612 192 KB

The r_hats of the deltas are all NAN but also the means of the original posteriors (“rates”) have all become almost similar.

I’m not sure how to interpret this. But I must have made an obvious mistake in the setup as the data should lead to some meaningful results unlike this.
The same happens when I do this
deltaAB = pm.Deterministic(‘deltaAB’,obs[1] - obs[0])
deltaAC = pm.Deterministic(‘deltaAC’,obs[2] - obs[0])
deltaAD = pm.Deterministic(‘deltaAD’,obs[3] - obs[0])
instead of
delta = pm.Deterministic(‘deltaAB’,obs - obs[0])

It is important to me to be able making calculations like a delta (B-A) or B/A-1. Could you please point me to where the error originates?

2

I would like to provide some additional information with diagnostic plots (traceplot and posteriors) but sadly I’m only allowed to post one image per post as a new user:

Help would be greatly appreciated.

3

Hi Zappageck, you should change the np.log and np.power to T.log and T.power respectively. This should speed up the sampling.

I think your delta definition is wrong. I presume you want the differences between the probabilities of each trial? In that case you want to take differences of the rates variable.

the above posts suggests using a different parameterisation for the ab prior. On my machine this example ran about 10x quicker than your one.

However your data each suggest a probability around 1% after more than 7000 trials. No matter what model you use the output rates can’t move too far from the frequentist approach.

2 Likes