# Average loss in ADVI optimization

**URL:** <https://discourse.pymc.io/t/average-loss-in-advi-optimization/1558>\
**Category:** Questions\
**Created:** [July 23, 2018, 3:33pm UTC](https://discourse.pymc.io/t/average-loss-in-advi-optimization/1558 "2018-07-23T15:33:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mkesin](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/mkesin/32/992_2.png) [@mkesin](https://discourse.pymc.io/u/mkesin)\
**Post date:** [July 23, 2018, 3:33pm UTC](https://discourse.pymc.io/t/average-loss-in-advi-optimization/1558/1 "2018-07-23T15:33:14Z")

</div>

I’ve run into a situation where the average loss first decreases with number of steps, then increases. Should I assume that the model deteriorated based on this? (something in me says “no” but I can’t justify the intuition)

---

<div class="post-metadata">

**Author:** ![bwengals](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/bwengals/32/1237_2.png) [@bwengals](https://discourse.pymc.io/u/bwengals)\
**Post date:** [July 23, 2018, 4:07pm UTC](https://discourse.pymc.io/t/average-loss-in-advi-optimization/1558/2 "2018-07-23T16:07:42Z")

</div>

What happens if you run for even more iterations? Does the avg. loss decrease again? You could consider fiddling with the learning rate or momentum parameters, perhaps turn them down.

---

<div class="post-metadata">

**Author:** ![mkesin](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/mkesin/32/992_2.png) [@mkesin](https://discourse.pymc.io/u/mkesin)\
**Post date:** [July 23, 2018, 5:32pm UTC](https://discourse.pymc.io/t/average-loss-in-advi-optimization/1558/3 "2018-07-23T17:32:26Z")

</div>

Thanks Bill - sorry for the dense question, having trouble finding learning rate param in the ADVI docs - would you mind pointing me in the right direction?

/m

---

<div class="post-metadata">

**Author:** ![bwengals](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/bwengals/32/1237_2.png) [@bwengals](https://discourse.pymc.io/u/bwengals)\
**Post date:** [July 23, 2018, 5:46pm UTC](https://discourse.pymc.io/t/average-loss-in-advi-optimization/1558/4 "2018-07-23T17:46:41Z")

</div>

Found a quick mention in the variational API quickstart, but it’s pretty easy to miss. You’ll set the `obj_optimizer` argument, for example:

```
with model:
    inference = pm.ADVI()
    approx = pm.fit(n=30000, method=inference, obj_optimizer=pm.sgd(learning_rate=0.01))

```

There are several stochastic minimization methods available, such as `pm.adam`, `pm.sgd`, `pm.adagrad`, `pm.adadelta`, and probably others. Each have different parameters you can set.