Calibrating an inverse geo test for a PyMC MMM model

Questions version agnostic
,
1

Hi everyone, :wave:

I’m building an MMM model using PyMC and need some help with calibrating an inverse geo test. Not sure if I set this up right, and I’d appreciate any guidance!

Test details:

  • Pre-test spend: $4500/day (whole market)
  • During test: $1700/day (control geos, which are amost unaffected), $0 (test geos)
  • Result: Δy = -400 installs in the test vs the control
  • Std deviation: 5000

Here’s the way I summarized it:

df_lift_test = pd.DataFrame({
    "channel": "channel_name",
    "x": 1700,
    "delta_x": -1700,
    "delta_y": -400,
    "sigma": 5000
})

Does this setup look right for calibrating the impact?

Sorry if this is a basic question - any feedback would be super helpful! :pray:

Thanks a lot!

2

Hey @A_Bell you may know already but here its a good description.

3

Hey @cetagostini, thanks! I’ve seen it, but I’m still unsure. My situation is a bit different since it’s inverse- I’ve lowered the budget. So I’m not sure if x and Δx should be for the entire market or just the test market.

The counterfactual thinking behind geo tests is that we created a test group where the budget was reduced to $0, and a comparable control group where the budget remained at $1700 (their pre-test allocation). So the difference in treatment between test and control is $1700, and it resulted in a -400 change in y.

I feel like I need to incorporate that somehow into the model. What do you think? Can you help me understand what to input into the model?

Thanks! BTW- your webinar on MMM was incredibly interesting and insightful, thanks for that!