Optimize Channel Distribution

Questions version agnostic
1

Our team has been working on a MMM model and comparing PyMC’s DelayedSaturatedMMM to Robyn’s MMM model. Both models have their strengths and weaknesses on our dataset, but I am a little confused with the budget optimization (DelayedSaturatedMMM.optimize_channel_budget_for_maximum_contribution) in PyMC. I understand it is a marked as “experimental” in the documentation and may still be a work in progress.

Here are my questions:

  • What number of period(s) is the budget optimization for? Based on my experience, it seems to be for 1 training period. If I trained on weekly data, the optimization is for a budget for one week. I cannot find any documentation to confirm these assumptions?
  • If it is one period, how do we extend optimization over multiple periods?
  • Adstock carryover is an important part of MMM. Is recent spend considered in the optimization method? I would assume it has to be, but I don’t see anywhere on the input that it would be.

I am more familiar with Robyn’s model and optimization so my questions may be skewed by that experience. Please feel free to correct me or ask for clarification.

Really what I want to understand his “what period(s) are the optimization for (past or historical?” and “does it consider recent spend when optimizing?”.

Thanks

1 Like
2

Hi @markgroner, Thank you for your questions. Let me try to add my input (maybe @cetagostini can provide more details).

Before getting into details, I strongly recommend referring to the example notebook at Budget Allocation with PyMC-Marketing — pymc-marketing 0.6.0 documentation, as there you can find a complete guide on the approach (see also the original post).

Now, back to your questions:

  • What number of period(s) is the budget optimization for? Based on my experience, it seems to be for 1 training period. If I trained on weekly data, the optimization is for a budget for one week. I cannot find any documentation to confirm these assumptions?

We are currently using the mmm fir from historical data to find an optimal budget distribution under constraints. That is, we look into the contribution curves:

image
image2423×1623 275 KB

and use a fit (sigmoid or michaelis-menten) to parameterize it to find an inflection point, which we use to compute the optimal mix.

The output of the optimization will be a new optimal global budget distribution so that you can run difference scenario-simulations:

image
image3023×1447 123 KB

  • If it is one period, how do we extend optimization over multiple periods?

We do not have an optimal mix per week. A possibility would be to run the procedure in a time-slice-cross validation way and see how the optimal distribution changes as we move the window forward.

Once we add time-varying parameters, we expect these time fluctuations to become even more meaningful. In many applications, the total budget distribution is decided monthly or quarterly (e.g. for TV, buying media requires time).

  • Adstock carryover is an important part of MMM. Is recent spend considered in the optimization method? I would assume it has to be, but I don’t see anywhere on the input that it would be.

The adstock effect is implicitly added in the contribution curves as we use the model output.

We are actively working on some internal refactor and we are revisiting and bouncing ideas on how to improve the optimizer (or even offer more ways of doing this). Please follow the discussion in Budget Allocation on a fitted MMM models · pymc-labs/pymc-marketing · Discussion #684 · GitHub

1 Like
3

Most of the internet sources are not very kind to the Robyn. Can you elaborate what strengths of Robyn your team has noticed? Typically researchers are initially very optimistic about Robyn but after investing few months into this tool they start noticing they get almost random results, they get hundreds of models with wildly differing implications with no clear indication which one to prefer. Try running same data and same model on few different starting seeds and each seed will produce you vastly different models.
I would be very interested in hearing your point of view and your experiences when it comes to working with Robyn.