# Binomial AB Testing

**URL:** <https://discourse.pymc.io/t/binomial-ab-testing/14279>\
**Category:** v5\
**Tags:** modeling\
**Created:** [April 17, 2024, 10:14pm UTC](https://discourse.pymc.io/t/binomial-ab-testing/14279 "2024-04-17T22:14:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jkenber](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@jkenber](https://discourse.pymc.io/u/jkenber)\
**Post date:** [April 17, 2024, 10:14pm UTC](https://discourse.pymc.io/t/binomial-ab-testing/14279/1 "2024-04-17T22:14:34Z")

</div>

Newbie here.

I’m trying to address a limitation in Bayesian AB tests. We’re assuming every observation is independent, but we often observe each customer more than once. I have a model that works on synthetic data, but it is very slow when realistic data volumes are provided (BetaBinomial is a poor fit, so I don’t want to rely upon it w/conjugates / Gibbs).

I think there is redundancy in the example below because the conversion likelihood for each individual customer is sampled in the posterior. But I can’t find a way to draw independent samples from a uniform prior and feed those into the Binomial p parameter. If I give the uniform prior no shape, the model will assume all customers have the same conversion likelihood, p.

```auto
import pymc as pm
import pandas as pd
import numpy as np

observations = [30]*20 #attempts, each entry represents a new customer 
occurrences = [2]*10 + [28]*10 #successes for the corresponding customers

with pm.Model() as model:

    p_t = pm.Uniform("test_distn", 0, 1, shape=(len(observations), ))
    p_t_mean = pm.Deterministic("test", p_t.mean())
    obs_test = pm.Binomial("obs_test", n=np.array(observations), p=p_t, observed=occurrences)

    step = pm.Metropolis()
    trace = pm.sample(posterior_sample + burn, chains=2, step=step)

```

I also end up randomly flattening “test\_distn” from the trace to see how the customers success likelihood is distributed. Which works, but feels ugly/hacky

This attempt to remove redundancy doesn’t work. Posterior for “test\_distn” is clearly wrong

```auto
import pymc as pm
import pandas as pd
import numpy as np

observations = [30]*20 #attempts, each entry represents a new customer 
occurrences = [2]*10 + [28]*10 #successes for the corresponding customers

with pm.Model() as model:

    p_t = pm.Uniform("test_distn", 0, 1)
    p_t_n = pm.draw(p_t, draws = len(observations))
    p_t_mean = pm.Deterministic("test", p_t.mean())
    obs_test = pm.Binomial("obs_test", n=np.array(observations), p=p_t_n, observed=occurrences)

    step = pm.Metropolis()
    trace = pm.sample(posterior_sample + burn, chains=2, step=step)

```

Any help would be most appreciated. It feels like an efficient solution here is highly applicable, hopefully others could benefit too

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [April 18, 2024, 11:58pm UTC](https://discourse.pymc.io/t/binomial-ab-testing/14279/2 "2024-04-18T23:58:27Z")

</div>

Welcome!

There are several things here that seem less than ideal. First, you should probably stick to calling `pm.sample()`, passing no arguments so that it executes using all the default values for all arguments (this is particularly true for the `step` argument). Because `observations` is always 30, you can don’t have to pass `n=np.array(observations)` into the binomial prior, you can just set `n=30`.

> [@jkenber](#):
>
> I also end up randomly flattening “test\_distn” from the trace to see how the customers success likelihood is distributed. Which works, but feels ugly/hacky

I’m not sure what you mean by “flattening” here. When I run the following model:

```python
n_flips = 30
observations = [n_flips] * 20
occurrences = [2] * 10 + [28] * 10

with pm.Model() as model:
    p_t = pm.Uniform("test_distn", 0, 1, shape=(len(observations),))
    obs_test = pm.Binomial("obs_test", n=n_flips, p=p_t, observed=occurrences)
    idata = pm.sample()

```

I inspect the parameter you asked about:

```python
idata.posterior["test_distn"].mean(dim=["chain", "draw"])

```

I get this:

```python
<xarray.DataArray 'test_distn' (test_distn_dim_0: 20)> Size: 160B
array([0.09349925, 0.09319441, 0.09354546, 0.09385241, 0.09396087,
       0.09368252, 0.09379583, 0.09246933, 0.09423234, 0.09350552,
       0.9054838 , 0.90666993, 0.90597412, 0.90627215, 0.90680455,
       0.9063937 , 0.90617682, 0.90684242, 0.90594251, 0.90559609])
Coordinates:
  * test_distn_dim_0 (test_distn_dim_0) int64 160B 0 1 2 3 4 ... 15 16 17 18 19

```

which seems about right to me.

Happy to answer questions. I might also suggest checking out [this notebook](https://nbviewer.org/github/cluhmann/DBDA-python/blob/master/Notebooks/Chapter%209.ipynb) for some ideas about how to attack these sorts of models in a slightly more general manner.

---

<div class="post-metadata">

**Author:** ![jkenber](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@jkenber](https://discourse.pymc.io/u/jkenber)\
**Post date:** [April 19, 2024, 9:29am UTC](https://discourse.pymc.io/t/binomial-ab-testing/14279/3 "2024-04-19T09:29:50Z")

</div>

Thanks for this. Really great to get advice from the community so quickly. It does run much quicker using default args, NUTS sampler, smaller sample size.

You asked what I meant by flattening.

I had been accessing the posterior samples using idata.posterior[“test\_distn”].data  
This has shape (chains, posterior\_sample\_size, n\_customers)

i.e. we’re learning the posterior for every customer individually which doesn’t seem necessary to me.  
Perhaps this is unavoidable.

To learn the distribution amongst customers I had been randomly selecting one customer from each posterior sample, reducing the shape to (chains, posterior\_sample\_size)

The extraction you’ve shown appears to do a similar thing:  
idata.posterior[“test\_distn”].mean(dim=[“chain”, “draw”])

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [April 19, 2024, 1:51pm UTC](https://discourse.pymc.io/t/binomial-ab-testing/14279/4 "2024-04-19T13:51:34Z")

</div>

> [@jkenber](#):
>
> i.e. we’re learning the posterior for every customer individually which doesn’t seem necessary to me.  
> Perhaps this is unavoidable.

I would encourage you to check out [the notebook](https://nbviewer.org/github/cluhmann/DBDA-python/blob/master/Notebooks/Chapter%209.ipynb) I linked to as it takes a hierarchical approach. That is, it learned something about each customer, but uses what it learns to simultaneously learn something about all customers (which, in turn, informs what you learn about each customer).