# Simulating AB test outcome with prior data from same test on different platform

**URL:** https://discourse.pymc.io/t/simulating-ab-test-outcome-with-prior-data-from-same-test-on-different-platform/12828
**Category:** version agnostic
**Tags:** modeling
**Created:** [August 30, 2023, 3:47pm UTC](https://discourse.pymc.io/t/simulating-ab-test-outcome-with-prior-data-from-same-test-on-different-platform/12828 "2023-08-30T15:47:46Z")
**Posts on this page:** 1
**Page:** 1

<div class="post-metadata">

### Author: ![pblub](https://avatars.discourse-cdn.com/v4/letter/p/7ba0ec/32.png) [@pblub](https://discourse.pymc.io/u/pblub)
#### Post date: [August 30, 2023, 3:47pm UTC](https://discourse.pymc.io/t/simulating-ab-test-outcome-with-prior-data-from-same-test-on-different-platform/12828/1 "2023-08-30T15:47:46Z")

</div>

6 moths ago i ran a test on platform A that had favorable results on a rate metric (conversion metric). I want to run a simulation that would show possible outcomes on platform B.

Things to know:

- Platforms A and B have different prices
- Measured lift on platform A
- I know that platform B has lower conversion rates in general
- I suspect that test lifts on platform B will be lower then measured lifts on platform A

Since A and B have different prices I want to calculate possible revenue before running the test to determine if it is worth to run the experiment. My PYMC is not on the level I would like to have it also statistics is not the best.

Here is a NumPy approach:

```python
import numpy as np 
import pandas as pd 
import seaborn as sns
import matplotlib.pyplot as plt

N = 10000
cr_B = 0.22 # conversion rate before test
lift_A = 0.15 # measured lift of the test on platform A

prc_A = 64
prc_B = 45 
 
INSTALLS = 50000

# The lift will can be modeled with a beta distribution 
# I use "small" values for alpha and beta to have more variance in the distribution of possible lifts
dist_lift = np.random.beta(22,78,N)

# keeping only the values that are smaller then the lifts on platform A since platform B has worst performance
dist_lift = dist_lift[dist_lift<=lift_A]

# resampling to get back to the original number of lift points 
dist_lift = np.concatenate((
    dist_lift, 
    np.random.choice(dist_lift,size=N-len(dist_lift))
    ))

# CONVERSION RATE FOR PLATFORM B
# I have tis data and im confident in it so I uses actual numners for the beta dist.
disr_cr_real = np.random.beta(950,4318,N)

# SIMULATION FOR PLATFORM B
# multiply the conversions of B with the distribution of lifts 
dist_cr_simm = disr_cr_real * (dist_lift +1)

# FROM CONVERSION TO USERS TO REVENUE
rev_real = (disr_cr_real * INSTALLS) * prc_B
rev_simm = (dist_cr_simm * INSTALLS) * prc_A

fig, ax = plt.subplots(figsize=(10,5))
sns.kdeplot(rev_real, ax=ax, label='Raal', shade=True)
sns.kdeplot(rev_simm, ax=ax, label='Simm', shade=True)
ax.legend()

```

Looks nice, problem is I don’t know if i can trust this approach?  
The conversion rate of platform B shouldn’t greater then that on platform A.  
Is there a better way get a distribution for the lift than with resampling a filtered beta distribution?  
How do i code it pymc?

Your help and ideas are much appreciated.

Thanks