Average loss in ADVI optimization

Questions
1

I’ve run into a situation where the average loss first decreases with number of steps, then increases. Should I assume that the model deteriorated based on this? (something in me says “no” but I can’t justify the intuition)

2

What happens if you run for even more iterations? Does the avg. loss decrease again? You could consider fiddling with the learning rate or momentum parameters, perhaps turn them down.

3

Thanks Bill - sorry for the dense question, having trouble finding learning rate param in the ADVI docs - would you mind pointing me in the right direction?

/m

4

Found a quick mention in the variational API quickstart, but it’s pretty easy to miss. You’ll set the obj_optimizer argument, for example:

with model:
    inference = pm.ADVI()
    approx = pm.fit(n=30000, method=inference, obj_optimizer=pm.sgd(learning_rate=0.01))

There are several stochastic minimization methods available, such as pm.adam, pm.sgd, pm.adagrad, pm.adadelta, and probably others. Each have different parameters you can set.

1 Like