# Pymc3 variational inference for multi-level logistic regression returning approximation equal to NaN

**URL:** <https://discourse.pymc.io/t/pymc3-variational-inference-for-multi-level-logistic-regression-returning-approximation-equal-to-nan/7224>\
**Category:** Questions\
**Created:** [April 11, 2021, 10:16pm UTC](https://discourse.pymc.io/t/pymc3-variational-inference-for-multi-level-logistic-regression-returning-approximation-equal-to-nan/7224 "2021-04-11T22:16:45Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zhangyi\_Fan](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/zhangyi_fan/32/3920_2.png) [@Zhangyi\_Fan](https://discourse.pymc.io/u/Zhangyi_Fan)\
**Post date:** [April 11, 2021, 10:16pm UTC](https://discourse.pymc.io/t/pymc3-variational-inference-for-multi-level-logistic-regression-returning-approximation-equal-to-nan/7224/1 "2021-04-11T22:16:45Z")

</div>

I am trying to use variational inference via ADVI to sample from a posterior with a lot of parameters. The data is simulated using the following code:

```auto
import math

def sigmoid(x):
  return 1 / (1 + math.exp(-x))

sigmoid_vec = np.vectorize(sigmoid)

side_cov = np.random.normal(0,1,size = (1000,20))
betas = np.random.normal(0,1,size = (20,30))
res = np.dot(side_cov,betas)
main_cov = np.random.normal(0,1,size = (1000,30))
thetas = sigmoid_vec(np.sum(np.multiply(res,main_cov), axis = 1))
obs_y = (thetas > 0.5).astype(int)

```

Using this model:

```auto
with pm.Model() as model:
  # priors
  tilda = pm.Normal('tilda', mu=0.01, tau=1,shape = (side_cov.shape[1], main_cov.shape[1]))

  tilda_print = tt.printing.Print("tilda")(tilda)

  beta = pm.Deterministic('beta', tt.dot(side_cov,tilda))

  theta = pm.Deterministic('p', pm.math.sigmoid((beta * main_cov).sum(axis=1)))

  y = pm.Bernoulli('y',p = theta, observed=obs_y)

with model:
  advi_fit = pm.fit(method='advi', n=30000)

```

The error I am receiving is this:

```auto
NaN occurred in optimization. 
The current approximation of RV `tilda`.ravel()[0] is NaN.
The current approximation of RV `tilda`.ravel()[1] is NaN.
The current approximation of RV `tilda`.ravel()[2] is NaN.
The current approximation of RV `tilda`.ravel()[3] is NaN.
The current approximation of RV `tilda`.ravel()[4] is NaN.
The current approximation of RV `tilda`.ravel()[5] is NaN.

```

I suspect that this may have something to do with the fact that there are 3000 tildas to be sampled, but would appreciate any guidance. I have tried making the priors on tilda more informative but it didn’t seem to work.

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [April 12, 2021, 12:57pm UTC](https://discourse.pymc.io/t/pymc3-variational-inference-for-multi-level-logistic-regression-returning-approximation-equal-to-nan/7224/2 "2021-04-12T12:57:48Z")

</div>

You are probably observing numerical issues due to the `sigmoid` evaluating to either 0 or 1, and having an observation in `obs_y` that is the opposite.

You can try to use the argument `logit_p` in the `pm.Bernoulli` to pass directly the probability in logit scale without having to do `pm.math.sigmoid`

```python
logit_theta = pm.Deterministic((beta * main_cov).sum(axis=1))
y = pm.Bernoulli('y', logit_p=logit_theta, observed=obs_y)

```

Or you can use the `pm.math.invlogit` instead of `pm.math.sigmoid`, which avoids a `theta` of 0 or 1. This will ensure stability at the cost of less precision, so I would recommend the first solution if it solves your problem.

```python
theta = pm.Deterministic(pm.math.invlogit(beta * main_cov).sum(axis=1))
y = pm.Bernoulli('y', p=theta, observed=obs_y)

```

---

<div class="post-metadata">

**Author:** ![Zhangyi\_Fan](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/zhangyi_fan/32/3920_2.png) [@Zhangyi\_Fan](https://discourse.pymc.io/u/Zhangyi_Fan)\
**Post date:** [April 12, 2021, 9:25pm UTC](https://discourse.pymc.io/t/pymc3-variational-inference-for-multi-level-logistic-regression-returning-approximation-equal-to-nan/7224/3 "2021-04-12T21:25:29Z")

</div>

Thank you so much!!!