Hello,
I’m new in PyMC3. I was working with the HAR dataset - UCI HAR. And I was trying to modify the code given in the PyMC3 documentation file - Bayesian Neural Net. I added one extra layer, each layer is having 561 neuron/computation unit and I used a categorical distribution at the output node. Next, when I try to execute the mini batch program, I always get INF ELBO loss. I have attached the screenshot of my modified network. If necessary, I can upload my python notebook also. Can anyone help me please?
I’ve had similar problems with the multinomial distribution when some of the class probabilities converge to 0. Try adding a small jitter to act_out (i.e.,)…
act_out = T.nnet.softmax(pm.math.dot(act_2, weights_2_out)) + 1.0e-7
Let me know if that works. Good luck!
During the whole training time, the average loss is still showing ‘inf’ but at the end of the training, I’m getting an ELBO loss.
That’s probably because at some point during training you get an inf loss, but unlike NaNs, inf won’t cause a numerical error so your model can actually keep training. If I’m wrong here an actual dev should correct me, but you should be okay.
Thank you for your suggestion. I’ve one more extra question regarding mini batch if you can answer. If I use the following mini batch program, does it take only 50 training items from the dataset once or it keeps taking 50 training items until it finishes using all the training example and fit the model, just like the regular Stochastic Gradient Descent algorithm?
minibatch_x = pm.Minibatch(X_train, batch_size=50)
minibatch_y = pm.Minibatch(Y_train, batch_size=50)
neural_network_minibatch = construct_nn(minibatch_x, minibatch_y)
with neural_network_minibatch:
approx = pm.fit(40000, method=pm.ADVI())
Thanks again.
Neither actually, it will keep taking 50 samples from the dataset using sample with replacement - internally it generate a random index like [10, 15, 10, 1, 200, …] to index from the full dataset (It can have the slice appear more than once as well)