# Divergences with unused variables

**URL:** <https://discourse.pymc.io/t/divergences-with-unused-variables/13283>\
**Category:** General\
**Created:** [November 14, 2023, 3:05pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283 "2023-11-14T15:05:24Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![JcSun](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jcsun/32/7389_2.png) [@JcSun](https://discourse.pymc.io/u/JcSun)\
**Post date:** [November 14, 2023, 3:05pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/1 "2023-11-14T15:05:24Z")

</div>

Newbie question here… Is it normal to get divergences even when just sampling from two distributions? what does it mean?

Here is the code I used:

```auto
import pymc as pm

with pm.Model() as model:
    mu = pm.Normal("mu", mu=0, sigma=20)
    sigma = pm.HalfNormal("sigma", sigma=20)

    idata = pm.sample()

```

and this is the output I get

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/e/e63d2f4731de27f9ffe976ee2ef0ab849d5abaf7.png)

Am I doing something wrong or this has to be expected? Even raising target\_accept=0.95 I still get at least one divergence.

I checked [this](https://discourse.pymc.io/t/unused-variable-causing-divergences/5770) question already, but the answer was to use the problematic distribution as a prior (rather than letting it “unused”, but this looks confusion to me, meaning I don’t see why we shouldn’t be able to sample from a “free” distribution) and setting the `mu, sigma` parameters to push if further from zero. Does it mean every time I have a high probability mass on one value, e.g. 0, I will get divergences?

Thanks in advance for any comment!

---

<div class="post-metadata">

**Author:** ![jessegrabowski](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jessegrabowski/32/5010_2.png) [@jessegrabowski](https://discourse.pymc.io/u/jessegrabowski)\
**Post date:** [November 14, 2023, 9:23pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/2 "2023-11-14T21:23:40Z")

</div>

See [these comments](https://discourse.pymc.io/t/why-does-mcmc-without-observations-have-difficulty-sampling/12809/4) by @aseyboldt. Priors aren’t necessarily easier to sample than posteriors via MCMC.

This example is quite interesting though. I think it’s just the half-normal; the fact that there are two independent distributions is (i think) not relevant. I guess the boundary of the half-normal isn’t easy for NUTS? This model samples fine:

```auto
import pymc as pm

with pm.Model() as model:
    mu = pm.Normal("mu", mu=0, sigma=20)
    signed_sigma = pm.Normal("signed_sigma", sigma=20)
    sigma = pm.Deterministic('sigma', pm.math.abs(signed_sigma))

    idata = pm.sample()

```

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [November 14, 2023, 10:04pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/3 "2023-11-14T22:04:27Z")

</div>

Yeah the problem is with the mode/large mass at the edge of the support space in the HalfNormal

---

<div class="post-metadata">

**Author:** ![JcSun](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jcsun/32/7389_2.png) [@JcSun](https://discourse.pymc.io/u/JcSun)\
**Post date:** [November 15, 2023, 5:03pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/4 "2023-11-15T17:03:38Z")

</div>

thank you very much for your answers @ricardoV94 and @jessegrabowski , they make perfectly sense. But I wonder. Having a mode at boundary (zero in this case) for a prior/posterior distribution should be a pretty common case. Take for example the non-centered 8-school problem formulation. \tau is sampled according to a halfCauchy distribution [here](https://am207.github.io/2018fall/wiki/gelmanschools.html):

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/3/390465ccd22087ae84203447d89caef24f3a0733.png)

and these are the posteriors of the parameters (I guess you can ignore variables other than tau in the picture)

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/e/e037457e97365782f4004dd1123ad964c6c690ab.jpeg)

Shouldn’t I end up with divergences then in this case (and all other similar cases)?

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [November 15, 2023, 5:05pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/5 "2023-11-15T17:05:34Z")

</div>

> [@JcSun](#):
>
> Shouldn’t I end up with divergences then in this case (and all other similar cases)?

Looks like you ended up with divergences. Even a single divergence is indicative of problems. Sometimes it may look fine but if you sample for longer/more chains/different random seed they show up.

In most cases those kind of priors are fine as long as the posterior doesn’t have a lot of mass at the boundary, since what’s being sampled is the posterior not the prior.

---

<div class="post-metadata">

**Author:** ![JcSun](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jcsun/32/7389_2.png) [@JcSun](https://discourse.pymc.io/u/JcSun)\
**Post date:** [November 15, 2023, 6:22pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/6 "2023-11-15T18:22:15Z")

</div>

Ok I see. Then I guess one can try a Half-Cauchy/Half-Normal prior distributions and, if divergences happen because of the posterior having a mode at the boundary, one can start looking into [boundary-avoiding kind of priors](https://statmodeling.stat.columbia.edu/2018/05/04/zero-excluding-priors-hierarchical-variance-parameters-improve-computation-full-bayesian-inference/). Thanks to your comment, I also noticed that actually both in the Stan and Pymc documentation about the non-centered formulations they keep getting “ **false positive** ” divergences. I guess something else I could do is to check whether the same applies in my case.

Let me know if I am overthinking/overcomplicating this, but it starts to make sense. Thanks again for your comments!

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [November 15, 2023, 6:47pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/7 "2023-11-15T18:47:32Z")

</div>

I think you’re on the right track.

However, what do you mean by false positive divergences?

---

<div class="post-metadata">

**Author:** ![jessegrabowski](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jessegrabowski/32/5010_2.png) [@jessegrabowski](https://discourse.pymc.io/u/jessegrabowski)\
**Post date:** [November 15, 2023, 7:01pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/8 "2023-11-15T19:01:32Z")

</div>

I’m not a stats expert by any means, but I always thought the half-cauchy priors on variance were silly and overkill. Same goes for exponential. Have you ever actually drawn samples from these? They’re nuts.

I’ve been defaulting to Gamma(2, x) for strictly positive variables for a while now, and I haven’t felt the need to look back. It avoids these problems by explicitly ruling out zero. Sometimes when it’s not possible, or when I know ahead of time the variance is going to be close to zero, I’ve had good success switching to modeling the precision instead of the variance. I don’t exactly know what PyMC does under the hood to convert between parameterizations, but the sampler seems to be happier going after big numbers in the tail then flipping them.

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [November 15, 2023, 7:03pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/9 "2023-11-15T19:03:00Z")

</div>

> [@jessegrabowski](#):
>
> I’ve been defaulting to Gamma(2, x) for strictly positive variables for a while now

I have also switched to this. The idea of nearly-zero scale parameters doesn’t seem sensible as a general rule.

---

<div class="post-metadata">

**Author:** ![JcSun](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jcsun/32/7389_2.png) [@JcSun](https://discourse.pymc.io/u/JcSun)\
**Post date:** [November 15, 2023, 7:45pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/10 "2023-11-15T19:45:26Z")

</div>

From [here](https://www.pymc.io/projects/docs/en/v3.11.4/pymc-examples/examples/diagnostics_and_criticism/Diagnosing_biased_Inference_with_Divergences.html)

> As shown above, the effective sample size per iteration has drastically improved, and the trace plots no longer show any “stickyness”. However, we do still see the rare divergence. These infrequent divergences do not seem concentrate anywhere in parameter space, which is indicative of the divergences being false positives.)

and similarly [here](https://mc-stan.org/users/documentation/case-studies/divergences_and_bias.html) :

> These infrequent divergences do not seem concentrate anywhere in parameter space, which is indicative of the divergences being false positives. As expected of false positives, we can remove the divergences entirely by decreasing the step size,

---

<div class="post-metadata">

**Author:** ![JcSun](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jcsun/32/7389_2.png) [@JcSun](https://discourse.pymc.io/u/JcSun)\
**Post date:** [November 15, 2023, 7:46pm UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/11 "2023-11-15T19:46:58Z")

</div>

yes, i tried sampling from the half-cauchy and I was surprised as well! 😅 thanks a lot for the comment, super interesting!

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [November 16, 2023, 12:55am UTC](https://discourse.pymc.io/t/divergences-with-unused-variables/13283/12 "2023-11-16T00:55:39Z")

</div>

We should probably avoid such language in the example notebooks (divergences always indicate that the sampler is having issues, but sometimes you might care more or less about these issues). Just the sort of thing we hope to work on at this week’s [docathon](https://discourse.pymc.io/t/pymc-docathon-elevate-open-source-documentation/13248)! Feel free to join in if you’re interested/available.