Hello, I’m trying way to express prior on the ‘gain’ of an A/B test. This is a “basic setting” , without prior : visitors = [10,10] success = [5,5] with pm.Model() as model: a = pm.Beta(‘a’,alpha=1,beta=1) b = pm.Beta(‘b’,alpha=1,beta=1) obs_a = pm.Binomial(‘obs_a’,n=visitors[0],p=a,observe…

What if you penalize b-a instead of (b-a)/a?

using (b-a) instead of (b-a)/a, only seems to look better : no diagnostic message and ‘cleaner posterior’, but It’s only because in fact it make a wider prior regarding the sampled value. (so it’s like using sd=1 or 10 where there is no diagnostic error message…) Changing with sd=0.01 (instead of …

The issue is the funnel; as a → 0, then the (b-a) becomes a stricter and stricter constraint that a ~= b; so steps that were valid for a ~ 0.5 aren’t nearly as valid for a ~ 0.1. As you noted, dropping the denominator eliminated the issue with sampling – but your posterior again had that long tail s…

Thanks @chartl , your alternate parametrisation is interesting. My first intuition is that even it presents the problem differently, it just changes what you sample. My version samples a & b, and adds constraint on g, Your model samples a & g, and adds a constraint on b. This is the code (using …

I just added more draws to the same model : with pm.Model() as model: a = pm.Beta(‘a’,alpha=1,beta=1) g = pm.TruncatedNormal('gain', mu=0, sd=0.1, lower=-1, upper=(1-a)/a) b = pm.Deterministic('b', a * (g + 1)) obs_a = pm.Binomial('obs_a',n=visitors[0],p=a,observed=success[0]) obs_b = pm.Binom…

The next step of the diagnosis is to use a plot_trace and a pairplot (between gain and a) to see what is going on – though it should be stated that gain is not directly sampled, but instead a transformed variable. I wonder if there’s a way to plot that, too…

Expressing a prior thru a Potential

Questions

chartl April 28, 2021, 3:20pm 6

Can you try with a higher number of samples and show the traces? I’m interested to see if there’s obvious auto-correlation or if the model is getting stuck in a mode. Junpenglao has mentioned that the 25% is quite aggressive compared to stan:

Topic		Replies	Views
Gelman's (alpha + beta)^-5/2 parametrization for beta prior in AB test Questions	7	1140	April 3, 2021
Posterior always at lower limit of uniform prior Questions	4	986	August 25, 2021
Expected Results with 90% Divergences - Hierarchical Bayesian Issue v5	2	194	April 25, 2024
Define constraint on the priors Questions	12	2165	April 19, 2018
How best to use a posterior sample as a prior for a future integration in another model? Questions modeling	15	2528	June 27, 2024

Expressing a prior thru a Potential

Related topics