Inf Average loss ADVI in Correlated Topic Model

Questions
1

Hi,

While implementing Correlated Topic Model with ADVI I get inf average loss. What does it mean? Now I’m unsure whether I’m implementing it in a correct way or not.

Here is my code:

import numpy as np
import pymc3 as pm, theano.tensor as t
import matplotlib.pyplot as plt
K = 4
V = 4 # number of words
D = 10 # number of documents

data = np.random.randint(V,size=(D,4))

alpha = np.ones((1, K))
beta = np.ones((1, V))
model = pm.Model()

mu = np.ones((1,K),dtype = np.float64)
cov = np.random.random_sample(size=(K,K))

Wd = [len(doc) for doc in data]
(D, W) = data.shape

def log_lda(theta,phi):
    def ll_lda(value):  
        dixs, vixs = value.nonzero()
        vfreqs = value[dixs, vixs]
        ll =vfreqs* pm.math.logsumexp(t.log(theta[dixs]) + t.log(phi.T[vixs]), axis = 1).ravel()
        return t.sum(ll) 
    return ll_lda

with model: 
    eta = pm.MvNormal('eta',mu = mu,cov = cov, shape = (D,K))
    theta = t.nnet.softmax(eta)
    phi = pm.Dirichlet("phi", a=beta, shape=(K, V))
    doc = pm.DensityDist('doc', log_lda(theta,phi), observed=data)   
with model:    
    inference = pm.ADVI()
    approx = pm.fit(n=10000,method= inference,callbacks=[pm.callbacks.CheckParametersConvergence(diff='absolute')])
   
#inference    
tr1 = approx.sample(draws=1000)
pm.plots.traceplot(tr1);    
pm.plot_posterior(tr1, color='LightSeaGreen');

plt.plot(approx.hist)

Can someone please guide me with this?
Help much appreciated.
Thanks

2

Average Loss = inf: 100%|██████████| 10000/10000 [00:06<00:00, 1497.31it/s]
Finished [100%]: Average Loss = nan

3

What is plt.plot(approx.hist) looks like?

4

It gives me nothing. Its blank.

5

Yeah there is definitely some problems of your model. You should double check the likelihood implementation, print the testpoint and model.check_test_point.

6

It gives me

eta -inf
phi_stickbreaking__ -15.01367190100603
doc -72.08730677823429

For eta :

Screenshot%20from%202018-07-24%2012-37-24
Screenshot from 2018-07-24 12-37-24.png821×234 33.7 KB

Using this I have used
eta = pm.MvNormal('eta',mu = mu,cov = cov, shape = (D,K))

Is there anything wrong with this?

7

This is not right - the reason is that your cov is not a positive definite matrix.

8

In the documentation provided, its not mentioned about the positive definiteness of the covariance matrix?

9

I guess the document is not explicit enough on that but the logp of MvNormal (or MvStudentT) only defined on covariance matrix that are positive definiteness.

10

Okay! Thanks. :slight_smile:
Also do you know any implementation of Stochastic Mixed membership Block Models in pymc3. I want to implement it using ADVI.