Combined Multiple Hypothesis Tests

Exploring hypothesis testing using Bayesian statistics using PYMC3, mainly for interpretability of statistics and multiple hypothesis testing. New to the topic and pymc, so apologies if missing something obvious.

I understand how one can compare each variant to another to identify head to head winners. But how do you compare a single variant against a combination of the other variants in order to identify the probability of each variant being the winner from all variants?

In the example below, v3 is obviously the winning variant, but how do I compare against a combination of v1 and v2?

import pymc3 as pm

n = 10000
obs_v1 = 200
obs_v2 = 220
obs_v3 = 300

with pm.Model() as model:  # context management
    # define priors
    prior_v1 = pm.Beta('prior_v1', alpha=2, beta=10)
    prior_v2 = pm.Beta('prior_v2', alpha=2, beta=10)
    prior_v3 = pm.Beta('prior_v3', alpha=2, beta=10)

    # define likelihood
    like_v1 = pm.Binomial('like_v1', n=n, p=prior_v1, observed=obs_v1)
    like_v2 = pm.Binomial('like_v2', n=n, p=prior_v2, observed=obs_v2)
    like_v3 = pm.Binomial('like_v3', n=n, p=prior_v3, observed=obs_v3)

    pm.Deterministic('difference_2_1', prior_v2 - prior_v1)
    pm.Deterministic('difference_3_1', prior_v3 - prior_v1)
    pm.Deterministic('difference_3_2', prior_v3 - prior_v2)

    trace = pm.sample(draws=50000, progressbar=True)

_ = pm.traceplot(trace[1000:])

_ = pm.plot_posterior(trace[1000:],
                  var_names=['difference_2_1', 'difference_3_1', 'difference_3_2' ],
                  ref_val=0, color='#87ceeb')
1 Like