Problems with Hierachical Bayesian Model - Advertising


#1

Hi guys,

Approved_Conversion	Impressions interest		
2	19	1727646
7	19	2622588
10	91	17989844
15	63	10745856
16	141	31799775
18	33	8646488
19	33	6083217
20	48	6899804
21	25	2833490
22	12	3965401
23	7	1836368
24	15	2256874
25	20	5251853
26	23	4868639
27	54	16352527
28	42	10959630
29	132	18768653
30	13	2191807
31	15	1074288
32	35	6455261
36	10	922928
63	34	8365640
64	27	5085460
65	19	1737547
66	4	893407
100	9	2023690
101	25	2960453
102	7	1160953
103	5	1921053
104	8	1412110
105	6	2656351
106	5	1592431
107	20	4482111
108	7	2763404
109	8	2980365
110	9	2434719
111	10	1490896
112	15	2324572
113	7	1830565
114	4	1066164

Is my data -

And my model is the following - but I get convergence errors - any suggestion?

from pymc3 import Model, sample, Normal, HalfCauchy, Uniform


def ad_model2(impressions, conversion):
   
    N = len(impressions)

    with pm.Model() as pooled_model:

        phi = pm.Uniform('phi', lower=0.0, upper=0.001)

        kappa_log = pm.Exponential('kappa_log', lam=1.5)
        kappa = pm.Deterministic('kappa', tt.exp(kappa_log))

        thetas = pm.Beta('thetas', alpha=phi*kappa, beta=(1.0-phi)*kappa, shape=N)
        y = pm.Binomial('y', n=impressions, p=thetas, observed=conversion)
    return pooled_model

#2

Any suggestions to improve this model? Seems to be having trouble plotting the forest plot of ‘theta’


#3

The prior is really narrow - could be problematic.


#4

How wide? 1.0e4


#5

I am not sure - maybe try doing a forward generation and check whether the generated data is in a reasonable range.


#6

I’m sorry how do I do a forward generation?


#7

Can’t help you with that (new feature, haven’t used yet, see commits ~1 month ago)

However:

  1. looking at your data, sometimes conversions > impressions, you need to cap that (but I presume you have)

  2. running this with a simple beta on the data above gives MAP of ~0.45, 0.3 for \alpha and \beta (using pandas):

    with pm.Model() as ad_model:
       alphabeta = pm.HalfFlat('alphabeta', shape=2)
       probs = pm.Beta('probs', alphabeta[0], alphabeta[1], shape=impressions.shape[0])
       conversion_rate = pm.Binomial('conv', p=probs, n=impressions['impressions'], observed=impressions.loc[:, ['impressions','conversions']].min(axis=1))
    

I’m not sure what you’re trying to accomplish by ‘pooling’ as you did. To reparametrize as a mean probability and a precision?

It might be interesting to model it as a mixture if you want to assign users to ‘high-converters’ or ‘low-converters’ segments.


#8

Well interest is different segments on Facebook. I wanted to pool them for that reason.

I’ll check the conversions etc. And make sure it’s capped.

Mixture model could be interesting!


#9

I don’t see in your code where you effectively pool. This would do it:

ninterests = len(interests);

with pm.Model() as ad_model:

   alphabeta = pm.HalfFlat('alphabeta', shape=(ninterests, 2))
   probs = pm.Beta('probs', \
             alphabeta[impressions['interests'].values(), 0], \
             alphabeta[impressions['interests'].values(), 1])
   pm.Binomial('conv', p = probs, \
                       n = impressions['impressions'], \
                observed = impressions.loc[:, ['impressions','conversions']].min(axis=1))

#10

Cool I see where I went wrong with this. I’ll fix this up.


#11

BTW, I think you could share \beta across groups, and pool \alpha's from a Gamma or exponential.

Mathematically, with N the number of groups interests, and i = 0,…,N-1 being the group to which user j belongs, U the number of users and j = 0,…,U-1 :

\theta = 1\\ \beta \sim HalfFlat()\\ \alpha_i \sim Exponential(\theta)\\ p_j \sim Beta(\alpha_i, \beta)\\ x_j \sim Binomial(p=p_j, n=y_j)

Adjust \theta (and /or try a Gamma instead of Exponential) according to what the posteriors and PPC’s tell you, and depending on how sparse the groups are.


#12

I’ve got some trouble getting this to work. I’ll dig into it soon.

You’re assuming impressions[‘interests’] is a dataframe right?


#13

You’re right, I’ve skipped a few steps.

I’m presuming the data you posted above was converted to a pandas dataframe (called impressions), and interests would come from:

impressions['interests'] = impressions['interests'].astype('categories').cat.codes
interests = impressions['interests'].unique()

alternatively:

impressions['interests'] = impressions['interests'].astype('categories')

with pm.Model() as ad_model:

    alphabeta = pm.HalfFlat('alphabeta', \
                             shape=(len(impressions['interests'].cat.categories), 2))
    probs = pm.Beta('probs', \
                 alphabeta[impressions['interests'].cat.codes, 0], \
                 alphabeta[impressions['interests'].cat.codes, 1])
    pm.Binomial('conv', p = probs, \
                        n = impressions['impressions'], \
                 observed = impressions.loc[:, ['impressions','conversions']].min(axis=1))

#14

Uniform priors at log scale work nicely

import pymc3 as pm
import pandas as pd
import io

def ad_model(impressions, conversions):
    N = len(impressions)
    with pm.Model() as model:
        phi_log = pm.Uniform('phi_log', lower=0, upper=30)
        kappa_log = pm.Uniform('kappa_log', lower=-10, upper=30)
        
        alpha_log = pm.Deterministic('alpha_log', kappa_log-(phi_log/2))
        alpha = pm.Deterministic('alpha', 2**alpha_log)
        
        beta_log = pm.Deterministic('beta_log', kappa_log+(phi_log/2))
        beta = pm.Deterministic('beta', 2**beta_log)
        
        thetas = pm.Beta('thetas', alpha=alpha, beta=beta, shape=N)
        thetas_log = pm.Deterministic('thetas_log', log2(thetas))

        y = pm.Binomial('y', n=impressions, p=thetas, observed=conversions)
        return model

df = pd.read_table(io.StringIO(data))
with ad_model(df.impressions, df.conversions):
    trace = pm.sample(10000, tune=2000, progressbar=True)
    pm.traceplot(trace)

where

data = """
id	conversions	impressions		
2	19	1727646
7	19	2622588
10	91	17989844
15	63	10745856
16	141	31799775
18	33	8646488
19	33	6083217
20	48	6899804
21	25	2833490
22	12	3965401
23	7	1836368
24	15	2256874
25	20	5251853
26	23	4868639
27	54	16352527
28	42	10959630
29	132	18768653
30	13	2191807
31	15	1074288
32	35	6455261
36	10	922928
63	34	8365640
64	27	5085460
65	19	1737547
66	4	893407
100	9	2023690
101	25	2960453
102	7	1160953
103	5	1921053
104	8	1412110
105	6	2656351
106	5	1592431
107	20	4482111
108	7	2763404
109	8	2980365
110	9	2434719
111	10	1490896
112	15	2324572
113	7	1830565
114	4	1066164
"""