Average loss in ADVI optimization

I’ve run into a situation where the average loss first decreases with number of steps, then increases. Should I assume that the model deteriorated based on this? (something in me says “no” but I can’t justify the intuition)

What happens if you run for even more iterations? Does the avg. loss decrease again? You could consider fiddling with the learning rate or momentum parameters, perhaps turn them down.

Thanks Bill - sorry for the dense question, having trouble finding learning rate param in the ADVI docs - would you mind pointing me in the right direction?


Found a quick mention in the variational API quickstart, but it’s pretty easy to miss. You’ll set the obj_optimizer argument, for example:

with model:
    inference = pm.ADVI()
    approx = pm.fit(n=30000, method=inference, obj_optimizer=pm.sgd(learning_rate=0.01))

There are several stochastic minimization methods available, such as pm.adam, pm.sgd, pm.adagrad, pm.adadelta, and probably others. Each have different parameters you can set.

1 Like