Budget recommendations don't align with saturation curves

I have a model fitted on five channels. The saturation curves (see attached image) all look fairly straight and linear

without any noticeable saturation. Channel A, the orange curve, has the highest ROAS and slope. I’m running my budget optimizer with

allocation_strategy, optimization_result = mmm.optimize_budget(response_variable=‘total_contribution’,budget=total_budget,num_periods=num_optimization_periods,budget_bounds=budget_bounds,minimize_kwargs={“method”: “SLSQP”,“options”: {“ftol”: 1e-9, “maxiter”: 5000},},)

And then getting the expected target contributions with

response = mmm.sample_response_distribution(
    allocation_strategy=allocation_strategy,
    time_granularity=model_granularity,
    num_periods=num_optimization_periods,
    noise_level=0.05,
)

The model is pushing my spend down for channel A drastically, to about 2/3 its original spend. That creates a total contribution across all my channels with the optimized sales that is significantly lower than if I had just naively spent the avg spend of the training data. Why is the optimizer significantly under spending my most efficient channel giving me an optimized budget that performs worse than my naive spend scenario?

cc @cetagostini

@fountainpen

  1. Are you comparing similar periods?
  2. Are you checking the lag effect of each channel?
  3. What version are you using?
  4. Are you working with mmm.multidimensional API or regular MMM api?

Without any of those, I can’t properly give you a better input. But looking at the saturation functions only is not enough to make the statement that “optimizer significantly under spending my most efficient channel“.

You can have channel with a long adstock effect, meaning their compose effect overtime can be better than your more “linear“ direct channel (if the lag effect for this one is low). On top of it, the optimizer distribute spend evenly across channels, if this assumption is different to your historical data (you never spend evenly every day) then results can change.

Note: We have a parameter which allow you to use a spend distribution over time similar to your historical spend, this could be a use case for it. On the other hand, you can simply sample different response distributions at different spend levels on the allocation strategy to build the precise curve use by the optimizer given a spend of X over N period of time, that response will consider the full function, will not be a only saturation parameter representation.

The best you can do to assess correctly, will be to take your df for each channel calculate the avg spend per channel, build a allocation_strategy Xarray with the information, and then set num_optimization_periods to len(df). Doing so, you then run sample_response_distribution and check if the response is higher (if you are using the objective to maximize the mean posterior, the response from this should be lower).

If that happens, then would be amazing if you can share a reproducible example!

Hope this brings light :light_bulb:

Thanks for the quick reply! This is very helpful. Attached is what my adstock decays look like using samples = mmm.adstock.sample_curve(parameters=mmm.fit_result). All my channels follow a similar decay rate. I’m using 0.15.1 with the regular MMM api, not the multidimensional one (I’m using the end to end example notebook as a guide).

For the following statement, I’m not sure I understand. Does that mean if my weekly budget is 1000 with 4 channels, each channel gets 250, and it attempts to optimize from there?

….the optimizer distribute spend evenly across channels, if this assumption is different to your historical data (you never spend evenly every day) then results can change.

So if I have budget bounds for one channel that are 300-400, then the optimization wouldn’t work correctly? You noted that there exists a parameter that works better for my use case, what is it? Is it budget_distribution_over_period? There’s no set spend distribution in my dataset so I am inclined to follow an even distribution of the recommendations.

In the end to end example notebook spend wasn’t evenly distributed every day and the regular MMM api optimizer was used. Here is the example I am following (excluding * len(channel_columns) as that seems to be a mistake in the documentation based on an earlier post I made).

mean_spend_per_period_test = (
    X_test[channel_columns].sum(axis=0).div(num_periods * len(channel_columns))
)

budget_bounds = {
    key: [(1 - percentage_change) * value, (1 + percentage_change) * value]
    for key, value in (mean_spend_per_period_test).to_dict().items()
}

For an input budget of 37615 per week across all my channels, the avg spend of each channel, an optimization period of len(df)=60 weeks, and noise level=0, I get a response of 0.676 per week. For an optimization period of 8 weeks I get 0.674.

Avg spends

{
    "channel A": 15510,
    "channel B": 7123,
    "channel C": 1841,
    "channel D": 4096,
    "channel E": 9045,
}

Budget bounds based on 50% flexibility from avg spend were:

channel A: $7,755 - $23,266, channel B: $3,562 - $10,685, channel C: $920 - $2,761, channel D: $2,048 - $6,144, channel E: $4,522 - $13,567

allocation_strategy, optimization_result = mmm.optimize_budget(response_variable=‘total_contribution’,budget=37615,num_periods=60,budget_bounds=budget_bounds,minimize_kwargs={“method”: “SLSQP”,“options”: {“ftol”: 1e-9, “maxiter”: 5000},},)
response_optimal = mmm.sample_response_distribution(
    allocation_strategy=allocation_strategy,
    time_granularity="weekly",
    num_periods=60,
    noise_level=0,
)

channel_contribution = response_optimal["channel_contribution"].mean(dim="sample").sum(dim="date") * mmm.get_target_transformer()["scaler"].scale_

Model was fit with GeometricAdstock(l_max=4, normalize=True), saturation=LogisticSaturation()

So in essence, it’s telling me that for 8 weeks, I should reallocate spend from googleads into other channels, but for 60 weeks I should put more spend into googleads. Does this make sense given my adstock and saturation curves? Thanks for all your feedback, this is super helpful for my learning on this awesome package!

@cetagostini

I upgraded to 0.16.0 and tried using the multidimensional api for budget optimization MultiDimensionalBudgetOptimizerWrapper (per the deprecation notice) but got an error when I tried to pass it my fitted model Model must have a 'dims' attribute. Are my old models incompatible with the multidimensional api? Is the deprecation warning only if I have a multi dimensional model?

Another thing I have noticed is that no matter how I set my num_periods or budget params in optimize_budget, if I use the default budget bounds of (0, total_budget) for each channel, I get an equal allocation of budget across all my channels. So if budget is 10000, then I get 2000 for five channels each. I believe that the optimizer might be getting stuck somewhere and can’t properly explore the solution space.

So in essence, it’s telling me that for 8 weeks, I should reallocate spend from googleads into other channels, but for 60 weeks I should put more spend into googleads

I would need to see your model, but that’s not crazy. It’s not the same to optimize to smaller periods where you can prefer channels with more direct impact and early saturation, than for longer periods were each channel composition effect can grow without saturation.

Thats precisely the reason why time is important. You could make as mentioned the comparison against the real spend and should bring clarity. So you have to allocations at 60 weeks.

On the other hand, you can play with

> budget_distribution_over_period : xarray.DataArray, optional

Regarding optimizer getting stuck, It can be several things, I would say:

1. Scales are off, you can quickly create an objective function with `original_function() * 1e10` and see how optimizer reacts. If after doing this the iterations increase, the problem was your response and input scales are quite far in magnitude, and the **augmented** objective was failing under the tolerance because the tiny gradient given the difference in magnitude from both scales.

2. Current gradient based on your function is not smooth, makes SLSQP struggle (It may not be the best given your posterior). Then write the new smooth objective or changing the algorithm could be an option, but can’t say more without see specific output, and posterior responses.

Right now, if you are moving to multidimensional you need to train models again. Old models from previous API are not compatible indeed.

@cetagostini

Hmmm. I retrained my model using version 0.16.0 with mmm.fit() using the base MMM class but it’s still not compatible with MultiDimensionalBudgetOptimizerWrapper with the ‘dims’ error warning. Is there a different fit and MMM class I should be using? Should I only be using the MultiDimensionalBudgetOptimizerWrapper if I actually have multi dimensions despite the deprecation warning?

Model fitted with:

from pymc_marketing.mmm import MMM, GeometricAdstock, LogisticSaturation

mmm = MMM(
    model_config=model_config,
    sampler_config=sampler_config,
    date_column="date",
    adstock=GeometricAdstock(l_max=4, normalize=True),
    saturation=LogisticSaturation(),
    channel_columns=channel_columns,
    control_columns=control_columns,
    yearly_seasonality=1,
    validate_data=True,
)

mmm.fit(
    X=X_train,
    y=y_train,
    target_accept=0.99,
    chains=4,
    draws=4000,
    tune=3000,
    nuts_sampler="pymc",
    random_seed=rng,
)

mmm.save("pymcm_controls.nc")

from pymc_marketing.mmm.multidimensional import (
    MultiDimensionalBudgetOptimizerWrapper,
)

mmm = MMM.load("pymcm_controls.nc")

optimizable_model = MultiDimensionalBudgetOptimizerWrapper(
    model=mmm, start_date="2024-09-06", end_date="2024-11-29"
)

Hey @fountainpen

Yes, MMM will be deprecated soon. Can you try `from pymc_marketing.mmm.multidimensional import MMM` and train your models with this class? Instead of regular MMM class? Compatibility between regular MMM and multi dim optimizer is not built or planned.