MultiDimensional MMM: Drastically Different Performance from Old Class

1

Hello!

When I’m using the multidimensional MMM, I’ve noticed a much lower MAPE and different channel contributions. Is there a reason why?

For context, my MMM is hierarchical at a channel level, but predictions are not meant to be broken out according to geo/other expected dimensions.

For reference, I’ve provided my old & new code snippets below:

New:

saturation = LogisticSaturation(
    priors={
        "beta": saturation_beta, #Hierarchical Priors Based on Platform ,dims = 'channels'
        "lam": Prior(
            "Gamma",
            mu=0.5,
            sigma=0.25,)}
)

adstock = GeometricAdstock(
    priors={"alpha": adstock_alpha}, #Hierarchical Priors Based on Platform ,dims = 'channels'
    l_max=13,
)

custom_model_config = {
    "intercept": Prior("Normal", mu=0.2, sigma=0.02), #0.25
    "gamma_control": Prior("HalfNormal", sigma=0.02, dims = 'control'),
    "gamma_fourier": Prior("Laplace", mu=0, b=0.2),
}

mmm = MMM(
    date_column="week",
    target_column=KPI,
    channel_columns=sorted(paid_media_vars + organic_media_vars),
    control_columns=control_vars,
    scaling={
        "channel": {"method": "max", "dims": ()},
        "target": {"method": "max", "dims": ()},
        "control": {"method": "max", "dims": ()}
    },
    adstock=adstock,
    saturation=saturation,
    yearly_seasonality=4,
    model_config=custom_model_config,
)

Old:

saturation = LogisticSaturation(
    priors={
        "beta": saturation_beta, #Hierarchical Priors Based on Platform ,dims = 'channels'
        "lam": Prior(
            "Gamma",
            mu=0.5,
            sigma=0.25,)}
)

adstock = GeometricAdstock(
    priors={"alpha": adstock_alpha}, #Hierarchical Priors Based on Platform ,dims = 'channels'
    l_max=13,
)

custom_model_config = {
    "intercept": Prior("Normal", mu=0.2, sigma=0.02), #0.25
    "gamma_control": Prior("HalfNormal", sigma=0.02),
    "gamma_fourier": Prior("Laplace", mu=0, b=0.2),
}

from sklearn.preprocessing import MaxAbsScaler
scaler = MaxAbsScaler()
df[control_vars] = scaler.fit_transform(df[control_vars])

mmm = MMM(
    model_config=custom_model_config,
    sampler_config={"progressbar": True},
    date_column="week",
    adstock=adstock,
    saturation=saturation,
    channel_columns=sorted(paid_media_vars + organic_media_vars),
    control_columns=control_vars
    yearly_seasonality=4 
)
1 Like
2

Likely because the new MMM has pooled adstock/saturation parameters across channels by default

The next release will have reverted that, but in the meantime you can check what this guide suggested MMM API Migration Guide — Open Source Marketing Analytics Solution

3

Would the pooling still apply even though I’m setting my saturation and adstock using hierarchical priors (commented in the code I’ve shared above)?

4

It would apply to the parameters you are not specifying. You can do mmm.table() (or something like that) to get a view of the parameters involved and their dims. Then compare with the old MMM

5

Got it, thank you so much! I’ve changed my new model to replicate my old parameters. I’m still noticing some minor changes in channel contribution % - could this be due to underlying changes in the new MMM class?

6

Are they systematic, or could just be sampling noise?

7

They’re extremely minor (ex. % contribution changes by .1% - .3%), so I would assume it’s sampling noise, if the changes made to the MMM class shouldn’t result in any changes, but wanted to make sure!