Inconsistent ROAS when migrating to multi-dim MMM class

Questions v5
1

Hello,
I trained an MMM using pymc-marketing and realized that it’s probably better to migrate to the multi-dim MMM class for budget allocation analysis, even though I have a single geo to model. Despite keeping everything the same (input data, model configs, random seed for model fit), I see a significant difference in the ROAS between the two models. In my understanding, the results matched better with the real-world scenario for the single geo MMM. I don’t understand what could have caused this — looking for advice. Please let me know if I should provide more context. Thank you.

prior_sigma = total_spend_per_channel / total_spend_per_channel.sum()

my_model_config = {
“intercept”: Prior(“HalfNormal”, sigma=5, dims=(“geo”,)),
“saturation_beta”: Prior(“HalfNormal”, sigma=prior_sigma, dims=(“channel”,)),
“gamma_control”: Prior(“Normal”, mu=0, sigma=0.05, dims=(“control”,)),
“gamma_fourier”: Prior(“Laplace”, mu=0, b=0.2, dims=(“geo”, “fourier_mode”)),
“likelihood”: Prior(“TruncatedNormal”, sigma=Prior(“HalfNormal”, sigma=6), dims=(“date”, “geo”)),
}

my_sampler_config = {“progressbar”: True}

mmm = MMM(
date_column="date",
target_column="y",
channel_columns=channels,
control_columns=controls,
scaling={
“channel”: {“method”: “max”, “dims”: ()},
“target”: {“method”: “max”, “dims”: ()},
},
adstock=GeometricAdstock(l_max=10),
saturation=LogisticSaturation(),
yearly_seasonality=2,
model_config=my_model_config,
sampler_config=my_sampler_config,
dims=("geo",),
)
2

An important difference is that the multidimensional MMM defaults to pooled modeling where no explicit dimensions are specified, whereas the single dimensional MMM defaults to separate model.

Looking at your model_config, you might want to add in priors with dims=”channel” for adstock_alpha, and saturation_lam. This dhould unify the model to what you had with the single dimensional class.

3

Hello,

Thanks for writing, I tried adding dims argument in Prior for adstock_alpha and saturation_lam in my_model_config and it looks like the following

my_model_config = {

    "intercept": Prior("HalfNormal", sigma=5),

"saturation_beta": Prior("HalfNormal", sigma=prior_sigma, dims=("channel")),

"saturation_lam": Prior("Gamma", alpha=3, beta=1, dims=("channel")),

"adstock_alpha": Prior("Beta", alpha=1, beta=3, dims=("channel")),

"gamma_control": Prior("Normal", mu=0, sigma=0.05, dims=("control")),

"gamma_fourier": Prior("Laplace", mu=0, b=0.2, dims=("geo","fourier_mode")),

"likelihood": Prior("TruncatedNormal", sigma=Prior("HalfNormal", sigma=6), dims=("date")),

}

I could see improvement in low spend channels, but for big channels ROAS are high and not matching with the regular mmm. Multi-d MMM gives the following

{'Channel-a': 9.21,
'Channel-b': 0.22,
'Channel-c': 0.76,
'Channel-d': 1.11,
'Channel-e': 0.86,
'Channel-f': 1.74,
'Channel-g': 1.07,
'Channel-h': 2.33,
'Channel-i': 0.185}

and ROAS from regular MMM are as follows

{'Channel-a': 2.13
'Channel-b': 0.82,
'Channel-c': 0.56,
'Channel-d': 1.71,
'Channel-e': 0.89,
'Channel-f': 1.23,
'Channel-g': 0.15,
'Channel-h': 1.89,
'Channel-i': 0.38}

In my case, ‘Channel-a’ is the one with highest spend share of 50% and multi-d MMM is really off in predicting its ROAS (with a huge credible interval). I also tried adding the ‘geo’ argument in dims but it didn’t help either. To add more context, prior_sigma which in my case is spend share is the following

array([0.49728545, 0.08402681, 0.10286331, 0.0533288 , 0.02376035,
       0.06204508, 0.15416075, 0.01652881, 0.00600064])
4

I figured out the issue, in the new multi MMM class, xarray channel coordinate in model’s inference data (idata) are in alphabetic order whereas in the older MMM class it was in the same order as defined in channels column. This created a problem when I was using the same code for calculating ROAS (contribution.sum()/spend.sum()) from regular MMM in new multi MMM. So in short for e.g. I was accidentally dividing channel A’s contribution from channel B’s spend sum.

5

I’d open an issue about that if I were you, seems like surprising behavior

6

sure, that’s a good idea. I have created one.