# Convergence problems with ADVI

**URL:** <https://discourse.pymc.io/t/convergence-problems-with-advi/1103>\
**Category:** Questions\
**Created:** [April 19, 2018, 6:57am UTC](https://discourse.pymc.io/t/convergence-problems-with-advi/1103 "2018-04-19T06:57:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![gokl](https://avatars.discourse-cdn.com/v4/letter/g/3da27b/32.png) [@gokl](https://discourse.pymc.io/u/gokl)\
**Post date:** [April 19, 2018, 6:57am UTC](https://discourse.pymc.io/t/convergence-problems-with-advi/1103/1 "2018-04-19T06:57:51Z")

</div>

Hi,

I am using minibatch ADVI to fit models for my use case, but for some reason I often very strange convergence/hist plots like this:

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/1X/7dbd4d2c0c25128e9baa74dffcbc88c739a3dbc3.png)

The elbo seems to stabilize at 30000, be stable for a while and then starts to get very unstable from around 60000.

I would like to know if you can point me to generic reasons that can cause such hist plots and what I could do to tackle this problem (I am new to the world of Bayesian modeling).

I would prefer not to show my model, as I got this problem with different models, so I was thinking the problem must be something more generic.  
I am using hierarchical models and one thing that I saw is that when certain groups only have very small samples (1 or 2) they tend to be estimated very wrong.

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [April 19, 2018, 8:33am UTC](https://discourse.pymc.io/t/convergence-problems-with-advi/1103/2 "2018-04-19T08:33:18Z")

</div>

My guess is that some RVs have gradient problem after a while.  
A few generic solutions:

1. trace the parameters at each iteration and see which one is causing the problem, see [http://docs.pymc.io/notebooks/variational\_api\_quickstart.html#Tracking-parameters](http://docs.pymc.io/notebooks/variational_api_quickstart.html#Tracking-parameters)
2. use more monte carlo sample to approximate the gradient: eg setting `obj_n_mc=25` or higher (default is 1)
3. try different optimizer with different learning rate, eg `obj_optimizer=pm.adagrad(learning_rate=.01)`

---

<div class="post-metadata">

**Author:** ![gokl](https://avatars.discourse-cdn.com/v4/letter/g/3da27b/32.png) [@gokl](https://discourse.pymc.io/u/gokl)\
**Post date:** [April 19, 2018, 3:08pm UTC](https://discourse.pymc.io/t/convergence-problems-with-advi/1103/3 "2018-04-19T15:08:18Z")

</div>

Using a different obj\_optimizer did the trick, to be specific adadelta with the defaults is providing very stable convergence results.  
Thank you for pointing me there.