I’m following the examples in order to do a POC for my job. I was doing fine until doing budget allocation. The allocate_budget_to_maximize_response functions has been running for hours and hasn’t given any results.
The data consist of three media channels spanning 7 months of daily data.
Here are the code snippets where I define the functions and its arguments
scaler = 1000
total_budget = 1000/scaler # Imagine is 5K or 5M
# Define your channels
channel_list = ["BRAND", "DEVOLUCION", "SEGURO DE VIDA"]
# The initial split per channel
budget_per_channel = total_budget / len(channel_columns)
# Initial budget per channel as dictionary.
initial_budget_dict = {channel: budget_per_channel for channel in channel_columns}
# bounds for each channel
min_budget, max_budget = 100/scaler, 1000/scaler
budget_bounds = {channel: [min_budget, max_budget] for channel in channel_columns}
hey @TheJarmanitor ! i’m running into the same issues with the 0.9.0 version as well…I’m running it on python 3.11 and with 16 vCPUs and 128GB memory… In the 0.8.0 version, i was able to run it perfectly fine… Have you found any solution to this? Would you mind sharing what machine you are running it on?
cc: @cluhmann@juanitorduz to shed some light on this too thanks very much!
TL;DR: I have the same issue on 0.10.0 and the adstock transformation in the objective seems to be the culprit, as it is sometimes fast and sometimes stuck for several seconds, without any apparent qualitative differences between its inputs.
Sometimes, within the same optimization run, the program spends many seconds on this line and sometimes it finishes in milliseconds. It varies unpredictably from iteration to iteration even for the same channel. Kind of surprising, since the length of the transformed vector is always the same (l_max + num_periods). So the only thing that I can speculate is that it’s due to properties of the transformed vector, like sparseness; e.g. in an iteration with very little spend maybe some elements are rounded to zero and the transformation can ignore those elements (long shot, I know).
Any help would be very appreciated, as we cannot really generate any results as it stands. Thanks in advance!
(A related quesiton is whether the optimization process and the required batched convolution is supposed to engage multiple cores; I can test with pytensor in the same python session that most of the time it uses all of my machine’s cores, but I see only one active core when running the budget allocation)
Update: The bottleneck appears to be the adstock, the saturation transform is fast (I decomposed the two in the objective function). The “slow” input vectors that are transformed also look similar in composition to the “fast” input vectors, so inpout difference are unlikely to be the reason.
Update 2: The adstock transformation getting “stuck” is completely data independent. I have modified the objective such that (just) the adstock transformation is immediately repeated when the first try takes longer than 5 seconds. One run took over 40 seconds, while 100 identical calls to the function immediately afterwards each took less than 0.5 seconds. Further decomposing the operations showed that what takes so long (sometimes) is the batched convolution, as expected.
After much bug hunt, including recreating the environment on a different VM and a custom implementation of allocate_budget (to no avail), this fixed the issue for me:
It seems to marginally increase the runtime of all adstock calls by maybe 20% but so far appears to prevent pytensor from getting stuck for minutes at a time and therefore makes the optimization feasible again!
thanks very much!