# Perplexity of a model

**URL:** <https://discourse.pymc.io/t/perplexity-of-a-model/25>\
**Category:** Questions\
**Tags:** from\_github\
**Created:** [June 13, 2017, 1:19pm UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25 "2017-06-13T13:19:34Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [June 13, 2017, 1:19pm UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/1 "2017-06-13T13:19:35Z")

</div>

[Github issue #2297](https://github.com/pymc-devs/pymc3/issues/2297) by [pwl](https://github.com/pwl):

I noticed that in some papers people determine whether the model converged based on [perplexity](https://en.wikipedia.org/wiki/Perplexity#Perplexity_of_a_probability_model) on some data set aside earlier for testing. Is there a way to compute the perplexity of a model with the new OPVI API? As far as I understand, to get the perplexity we need to compute the values of `log(q)` of the current variational approximation `q` on user-provided data.

I had a look at the source code and it seems it would amount to evaluating `logq` (or `norm_logq`, I’m not sure at this point) but the complexity of the OPVI API is beyond me at this point and I have no idea how to supply user data to these functions. Maybe I’m looking in the wrong place and there is an easier way around it.

Long term I hope to implement a perplexity-based convergence test as a callback.

Are the other techniques of evaluating the convergence, apart from simply looking at average loss, available in pymc3? For example, the Wikipedia article also mentions using cross-entropy as the test function.

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [June 13, 2017, 1:24pm UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/2 "2017-06-13T13:24:29Z")

</div>

## Replies:

[junpenglao](https://github.com/junpenglao): You can find the convergence evaluation callback for VI in PyMC3 here [https://github.com/pymc-devs/pymc3/blob/master/pymc3/variational/callbacks.py#L31](https://github.com/pymc-devs/pymc3/blob/master/pymc3/variational/callbacks.py#L31)  
So far we are doing it in by looking at the change of loss in a given window.

As for evaluating the values of `log(q)` maybe @ferrine knows a quick way to do that.

–  
[pwl](https://github.com/pwl): @junpenglao Doesn’t that callback simply look track a change in the norm of the overall parameter vector of approximations `q_i` (so means and stdevs of transformed probability distributions) instead looking at the loss? In particular, it is completely ignoring `loss` in its ` __call__ ` definition.

–  
[ferrine](https://github.com/ferrine): @pwl Yes, We are currently looking at parameters as it works for both parametric gradient and non parametric gradient descent.  
I hope I correctly understood the main idea of the proposed method. Evaluating likelihood for approximate model is quite easy. You can get access to `approx.model.observed_RVs` in callback and in the first iteration compile a function using `approx_lik = approx.apply_replacements(likelihood)`. Note that likelihood is scaled to the full dataset there if using minibatches

–  
[pwl](https://github.com/pwl): @ferrine thanks for the tip. How do I access `likelihood`? Is it stored somewhere in `approx`?

–  
[ferrine](https://github.com/ferrine): I think not yet. But you can make it

```python

# first, you create a tensor, that can be any expression that uses pymc3 random nodes
# in our case it is likelihood
likelihood = tt.sum([v.logpt for v in approx.model.observed_RVs])
# to evaluate it under approximate posterior you can replace random nodes with approximation
# for the only sample it is
approx_lik = approx.apply_replacements(likelihood)
# for many samples
approx_liks = approx.sample_node(likelihood, n_samples)
# then you can teke expectation under posterior.
marginal_lik = approx_liks.mean(0)
# after all, function can be compiled
marginal_lik_fn = theano.function([], marginal_lik)

```

---

<div class="post-metadata">

**Author:** ![ferrine](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ferrine/32/10_2.png) [@ferrine](https://discourse.pymc.io/u/ferrine)\
**Post date:** [June 13, 2017, 1:36pm UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/3 "2017-06-13T13:36:28Z")

</div>

Wow, nice, it will be great to continue discussion here

---

<div class="post-metadata">

**Author:** ![pwl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pwl/32/212_2.png) [@pwl](https://discourse.pymc.io/u/pwl)\
**Post date:** [June 13, 2017, 2:04pm UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/4 "2017-06-13T14:04:30Z")

</div>

Great! Thanks for setting this up @junpenglao!

@ferrine Thanks for a thorough answer. One thing I still don’t understand how can I supply custom data point for the `likelihood` to be evaluated at. I guess instead of computing the `mean(0)` I would have to evaluate each `logpt` at some point but I will take a closer look at that tomorrow.

---

<div class="post-metadata">

**Author:** ![ferrine](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ferrine/32/10_2.png) [@ferrine](https://discourse.pymc.io/u/ferrine)\
**Post date:** [June 13, 2017, 4:12pm UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/5 "2017-06-13T16:12:36Z")

</div>

data point where likelihood is evaluated can be a shared variable or just a symbolic tensor. There is no way to change it except using `theano.clone` and having a reference to your data. As a shortcut you can use `approx.sample_node(likelihood, n_samples, more_replacements={old_data_tensor:new_data_tensor})`

---

<div class="post-metadata">

**Author:** ![pwl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pwl/32/212_2.png) [@pwl](https://discourse.pymc.io/u/pwl)\
**Post date:** [June 27, 2017, 7:41am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/6 "2017-06-27T07:41:05Z")

</div>

@ferrine I am playing around with the new functionality of `sample_node` and `apply_replacements` and I’m slowly getting a hold of it (mostly thanks to your recent [blog post](https://ferrine.github.io/blog/2017/06/21/variational-api-quick-start/), having working examples to build on really helps). I still have some questions though, as some things are not entirely clear to me yet.

Why are there two functions `sample_node` and `apply_replacements` if the `sample_node` with `size=1` seems to do the same thing as `apply_replacements`?

When running a model with the `Tracker` callback involving `sample_node` or `apply_replacements` I’m getting much more cpu usage then without the callback (all 8 cores at 100%). Is that expected? Why?

---

<div class="post-metadata">

**Author:** ![ferrine](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ferrine/32/10_2.png) [@ferrine](https://discourse.pymc.io/u/ferrine)\
**Post date:** [June 27, 2017, 10:18am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/7 "2017-06-27T10:18:06Z")

</div>

More computations are done. That’s why you get cpuload

---

<div class="post-metadata">

**Author:** ![ferrine](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ferrine/32/10_2.png) [@ferrine](https://discourse.pymc.io/u/ferrine)\
**Post date:** [June 27, 2017, 10:22am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/8 "2017-06-27T10:22:56Z")

</div>

As for 2 methods: shapes differ. Moreover internally scan is behind `sample_node` while `apply_replacements` does not use them. I agree that is not the best design and it’s better to wrap up in single function 🙂

---

<div class="post-metadata">

**Author:** ![pwl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pwl/32/212_2.png) [@pwl](https://discourse.pymc.io/u/pwl)\
**Post date:** [June 27, 2017, 10:49am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/9 "2017-06-27T10:49:36Z")

</div>

Maybe I was not entirely clear with the CPU load. I’m doing inference on a GPU with usually just one core active so I was surprised to see such a dramatic increase in CPU usage. But that was my mistake: I have not wrapped the test data set with the `theano.shared` so these computations were done on a CPU instead on a GPU (as far as I understand that). It’s now working much better with the shared arrays.

---

<div class="post-metadata">

**Author:** ![pwl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pwl/32/212_2.png) [@pwl](https://discourse.pymc.io/u/pwl)\
**Post date:** [June 28, 2017, 8:17am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/10 "2017-06-28T08:17:53Z")

</div>

When running

```python
with pm.Model():
    val = pm.Poisson.dist(
        mu = pm.Gamma('U',1,1,shape=3)).logp(theano.shared(np.arange(3)))
    inference = pm.ADVI()
    inference.approx.sample_node(val)

```

I’m getting a [MethodNotDefined erorr](https://gist.github.com/pwl/7c393e722622e4395e48986c76d015a9). Am I doing something wrong here or is it a bug? Also the above snippet works perfectly fine if I remove `theano.shared` and use the numpy array directly.

To be clear, I’m using these flags:

```auto
os.environ['THEANO_FLAGS'] = 'device=cuda,floatX=float32,exception_verbosity=high'

```

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [June 28, 2017, 8:33am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/11 "2017-06-28T08:33:16Z")

</div>

Are you on master? The above code works fine for me.

---

<div class="post-metadata">

**Author:** ![pwl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pwl/32/212_2.png) [@pwl](https://discourse.pymc.io/u/pwl)\
**Post date:** [June 28, 2017, 9:09am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/12 "2017-06-28T09:09:42Z")

</div>

Yes, I am on the master (in particular [https://github.com/pymc-devs/pymc3/issues/2358](https://github.com/pymc-devs/pymc3/issues/2358) works for me already).

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [June 28, 2017, 9:13am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/13 "2017-06-28T09:13:47Z")

</div>

Oh I see your edit. It must be a GPU issue then…

---

<div class="post-metadata">

**Author:** ![pwl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pwl/32/212_2.png) [@pwl](https://discourse.pymc.io/u/pwl)\
**Post date:** [June 28, 2017, 9:42am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/14 "2017-06-28T09:42:42Z")

</div>

I posted an [issue](https://github.com/pymc-devs/pymc3/issues/2361) about that.

---

<div class="post-metadata">

**Author:** ![pwl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pwl/32/212_2.png) [@pwl](https://discourse.pymc.io/u/pwl)\
**Post date:** [June 28, 2017, 11:44am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/15 "2017-06-28T11:44:16Z")

</div>

Continuing on `sample_node` related errors:

```auto
data = np.random.randint(3,size=(2,2)).astype(np.float32)
with pm.Model():
    a = pm.Beta('a',1,1,shape=2)
    b = pm.Beta('b',1,1,shape=2)
    psi = theano.tensor.outer(a,b)
    theta = pm.Gamma('g',1,1,shape=(2,2))
    data = pm.ZeroInflatedPoisson('data',theta=theta,psi=psi,observed=data)

    inference = pm.ADVI()
    val = inference.approx.sample_node(
        pm.ZeroInflatedPoisson.dist(theta=theta,psi=psi).logp(data),size=1)
    inference.fit(n=2,
                  callbacks=[pm.callbacks.Tracker(val=val.eval)])

```

yields [`AttributeError`](https://gist.github.com/pwl/9bf9b660d7bd6b79b95570ead03b0c93) but trains fine if I remove the callbacks. I tried replacing `outer` with explicit multiplication but then it complains that `AttributeError: mul.0 has no test value`. It does not happen for scalar variables though (that is if `a` and `b` are scalars).

---

<div class="post-metadata">

**Author:** ![ferrine](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ferrine/32/10_2.png) [@ferrine](https://discourse.pymc.io/u/ferrine)\
**Post date:** [June 28, 2017, 12:25pm UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/16 "2017-06-28T12:25:23Z")

</div>

Are you able to reproduce the problem on master?

---

<div class="post-metadata">

**Author:** ![pwl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pwl/32/212_2.png) [@pwl](https://discourse.pymc.io/u/pwl)\
**Post date:** [June 28, 2017, 12:33pm UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/17 "2017-06-28T12:33:24Z")

</div>

Yes, I’m using the master branch (updated a few hours ago).

---

<div class="post-metadata">

**Author:** ![ferrine](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ferrine/32/10_2.png) [@ferrine](https://discourse.pymc.io/u/ferrine)\
**Post date:** [June 28, 2017, 2:55pm UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/18 "2017-06-28T14:55:44Z")

</div>

> [@pwl](#):
>
> callbacks=[pm.callbacks.Tracker(val=val.eval)]

You’d better define tracker before calling fit

---

<div class="post-metadata">

**Author:** ![ferrine](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ferrine/32/10_2.png) [@ferrine](https://discourse.pymc.io/u/ferrine)\
**Post date:** [June 28, 2017, 3:01pm UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/19 "2017-06-28T15:01:34Z")

</div>

I see the problem is hidden in missing test\_values somewhere in logp. That’s strange. I feel more worried about context dependent flag change

---

<div class="post-metadata">

**Author:** ![pwl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/pwl/32/212_2.png) [@pwl](https://discourse.pymc.io/u/pwl)\
**Post date:** [June 29, 2017, 7:26am UTC](https://discourse.pymc.io/t/perplexity-of-a-model/25/20 "2017-06-29T07:26:45Z")

</div>

One more thing, it only seems to happen only for `ZeroInflatedPoisson.dist(theta=theta,psi=psi)`, I tried using `Poisson.dist(mu=psi)` without any issues. And there is no difference between defining tracker outside of the call to `fit`, if that’s what you meant.

[Next page](https://discourse.pymc.io/t/perplexity-of-a-model/25.md?page=2)