Inf Average loss ADVI in Correlated Topic Model


While implementing Correlated Topic Model with ADVI I get inf average loss. What does it mean? Now I’m unsure whether I’m implementing it in a correct way or not.

Here is my code:

import numpy as np
import pymc3 as pm, theano.tensor as t
import matplotlib.pyplot as plt
K = 4
V = 4 # number of words
D = 10 # number of documents

data = np.random.randint(V,size=(D,4))

alpha = np.ones((1, K))
beta = np.ones((1, V))
model = pm.Model()

mu = np.ones((1,K),dtype = np.float64)
cov = np.random.random_sample(size=(K,K))

Wd = [len(doc) for doc in data]
(D, W) = data.shape

def log_lda(theta,phi):
    def ll_lda(value):  
        dixs, vixs = value.nonzero()
        vfreqs = value[dixs, vixs]
        ll =vfreqs* pm.math.logsumexp(t.log(theta[dixs]) + t.log(phi.T[vixs]), axis = 1).ravel()
        return t.sum(ll) 
    return ll_lda

with model: 
    eta = pm.MvNormal('eta',mu = mu,cov = cov, shape = (D,K))
    theta = t.nnet.softmax(eta)
    phi = pm.Dirichlet("phi", a=beta, shape=(K, V))
    doc = pm.DensityDist('doc', log_lda(theta,phi), observed=data)   
with model:    
    inference = pm.ADVI()
    approx =,method= inference,callbacks=[pm.callbacks.CheckParametersConvergence(diff='absolute')])
tr1 = approx.sample(draws=1000)
pm.plot_posterior(tr1, color='LightSeaGreen');


Can someone please guide me with this?
Help much appreciated.

Average Loss = inf: 100%|██████████| 10000/10000 [00:06<00:00, 1497.31it/s]
Finished [100%]: Average Loss = nan

What is plt.plot(approx.hist) looks like?

It gives me nothing. Its blank.

Yeah there is definitely some problems of your model. You should double check the likelihood implementation, print the testpoint and model.check_test_point.

It gives me

eta -inf
phi_stickbreaking__ -15.01367190100603
doc -72.08730677823429

For eta :

Using this I have used
eta = pm.MvNormal('eta',mu = mu,cov = cov, shape = (D,K))

Is there anything wrong with this?

This is not right - the reason is that your cov is not a positive definite matrix.

In the documentation provided, its not mentioned about the positive definiteness of the covariance matrix?

I guess the document is not explicit enough on that but the logp of MvNormal (or MvStudentT) only defined on covariance matrix that are positive definiteness.

Okay! Thanks. :slight_smile:
Also do you know any implementation of Stochastic Mixed membership Block Models in pymc3. I want to implement it using ADVI.