I have the following Deep Learning model: X = theano.shared(X_train) Y = theano.shared(np.where(y_train==1)[1]) X_off = theano.shared(X_offset_train) h = [n_features, 10, 10, 1] inits = [] for i in range(len(h)-1): inits.append(np.random.randn(h[i], h[i+1])* np.sqrt(2/(h[i] + …

[image] sachinruk: I understand that VI is different in that we are optimising the mu/ sigma of the approximate posterior (q(w)), so maybe we had a large mu value? I see, yes you can track the mu/sigma value, see http://docs.pymc.io/notebooks/variational_api_quickstart.html#Tracking-parameter…

Have you tried the methods to debug in this post? [image] Frequently Asked Questions Questions NaN occurred in optimization at the first iteration in VI (i.e., seeing something like 0%| | 0/5000 [00:00<?, ?it/s]) Check the initial approximation parameters: approx.approx.p…

I changed the question to mention that this didn’t happen in the very first iteration. I did get it running by using BoundedNormal = pm.Bound(pm.Normal, lower=-2.0, upper=2.0) instead of pm.Normal on the weights. However, again when I had put the limits to +/- 3 it still halted the inference process…

Ok, this is embarassing. Turns out the reason that it was stuffing up was because I had accidentally put in a relu at the last layer as well, instead of linear. Didn’t realise that having an activation in the output layer would have caused so much damage considering I’m still taking the softmax. Reg…

What do you mean by offending sample?

I’m assuming that when we were sampling the posterior, one of the samples had the value for one of the weights to be say 10000000. A ridiculously large number that somehow caused the log probability to be infinity. Similar argument for small values causing a small value through the softmax, causing …

NaN occurred in optimization (error in DL with variational minibatches)

Questions

junpenglao March 20, 2018, 7:56am 2

Have you tried the methods to debug in this post?

Topic		Replies	Views
NaN occurred in optimization with ADVI Questions	5	4677	August 6, 2019
NaN occurred in optimization at first Iteration with ADVI Questions	3	786	February 18, 2020
How to use pymc3.fit() method Questions	16	3903	May 7, 2018
ADVI changes start values to NaN Questions	2	920	January 18, 2021
Pymc3 variational inference for multi-level logistic regression returning approximation equal to NaN Questions	2	641	April 12, 2021

NaN occurred in optimization (error in DL with variational minibatches)

Related topics