Exploring hypothesis testing using Bayesian statistics using PYMC3, mainly for interpretability of statistics and multiple hypothesis testing. New to the topic and pymc, so apologies if missing something obvious.
I understand how one can compare each variant to another to identify head to head winners. But how do you compare a single variant against a combination of the other variants in order to identify the probability of each variant being the winner from all variants?
In the example below, v3 is obviously the winning variant, but how do I compare against a combination of v1 and v2?
import pymc3 as pm n = 10000 obs_v1 = 200 obs_v2 = 220 obs_v3 = 300 with pm.Model() as model: # context management # define priors prior_v1 = pm.Beta('prior_v1', alpha=2, beta=10) prior_v2 = pm.Beta('prior_v2', alpha=2, beta=10) prior_v3 = pm.Beta('prior_v3', alpha=2, beta=10) # define likelihood like_v1 = pm.Binomial('like_v1', n=n, p=prior_v1, observed=obs_v1) like_v2 = pm.Binomial('like_v2', n=n, p=prior_v2, observed=obs_v2) like_v3 = pm.Binomial('like_v3', n=n, p=prior_v3, observed=obs_v3) pm.Deterministic('difference_2_1', prior_v2 - prior_v1) pm.Deterministic('difference_3_1', prior_v3 - prior_v1) pm.Deterministic('difference_3_2', prior_v3 - prior_v2) trace = pm.sample(draws=50000, progressbar=True) _ = pm.traceplot(trace[1000:]) _ = pm.plot_posterior(trace[1000:], var_names=['difference_2_1', 'difference_3_1', 'difference_3_2' ], hdi_prob=0.89, ref_val=0, color='#87ceeb')