This is an issue I sometimes has as well, a few example on Github: FloatingPointError: NaN occurred in optimization with NUTS #2272 Issues getting variable in hierarchical model to update #2271 My first instinct is usually: model misspecification. For example, commenting on issue #2271 : I would…

New option after #2257 approx = pm.fit(n=10000, method='advi', model=model, obj_optimizer=pm.adagrad_window(learning_rate=2e-4), total_grad_norm_constraint=10 # other constants can appear here, I do not coin 10 )

Do we have an option for passing that into sample for advi initialization? And is there a possibility for the advi optimizer to ignore a couple of nans/infs near the start of the optimization?

Yes, start point is supported with start kwarg. Nan can’t be ignored as it goes to updates and breaks all. Infs can be ignored as they don’t turn updates to nan

Sorry, my question wasn’t very precise. What I meant was a way to pass parameters like the learning rate to pm.sample. Something analog to nuts_kwargs/step_kwargs. And about the nans: Wouldn’t it be possible to always store the previous state before a step and then if we encounter a nan go back one…

Hmm. Mass matrix start is not supported now for advi. Now advi in pm.sample uses defaults. Dealing with nans can be tricky. I see the only way in creating a custom callback that dumps parameters after every few steps

Is it difficult from an algorithmic point of view or because of the implementation? I think the “NaN occurred in optimization” errors are a major pain point for using NUTS right now. I’ve seen a couple of models that sampled well when using previous chains to initialize the scaling, but don’t work …

That’s not difficult. Just haven’t thought about it.

NaN occurred in optimization with NUTS

Questions

Topic		Replies	Views
ADVI changes start values to NaN Questions	2	922	January 18, 2021
NaN occurred in optimization at first Iteration with ADVI Questions	3	786	February 18, 2020
Transitioning from pm.sample() to pm.fit() v5	15	224	May 28, 2025
NaN occurred in optimization with ADVI Questions	5	4683	August 6, 2019
How to initialize NUTS sampler with advi using non-standard obj_optimizer? v5 variational_inferenc , sampling	3	688	April 14, 2023

NaN occurred in optimization with NUTS

Related topics