I am currently in the process of learning and experimenting with bayesian neural networks. I am trying to compare the results of a simple NN vs BNN.
Using the example at https://twiecki.io/blog/2016/06/01/bayesian-deep-learning/ I started by replacing the moons DS with any MNIST DS (fashion in this case).
I created a subset with 2 classes out of the 10 initial ones and I also scaled the images.
The code can be found here: https://github.com/nlucian/bayesian_neural_network_vs_nn/blob/master/bayesian_neural_networks02-for_online.ipynb
Unfortunately for some reason the Average Loss is stuck at inf/nan.
Average Loss = inf: 19%|█▊ | 46620/250000
The things I tried:
- using relu/tahn/sigmoid activation function.
- reducing the number of images to 500 before loading the images into the model
- tried multiple numbers for the hidden layers
Not really sure what the reason could be - even if I reduce the dimensionality to something absurd like 2 (as in the moons example) the Average Loss remains the same
Could you please help me with an advice? I have the example using [Lasagne] but I would like to know if it is possible to do it this way - for a better understanding of the inner workings before moving to something even better