Theano (lasagna) neural network not improved (learn at all)

Questions
1

Hi every one.
I want to predict time series numbers through a neural network and pymc.my simple dataset consist of sequence numbers which I want to predict fourth number (i.e. 10,15,20->25).
I have modeled it with pymc3 linear regression and also raw theano (lasagna) neural network (without pymc3). in order to use pymc and lasagna, I followed this tutorial except for regression goal, not classification. The issue is model not learn at all and predict almost same numbers of all inputs.
Here is my model:

def build_ann(init):
l_in = lasagne.layers.InputLayer(shape=(None,3),
input_var=input_var)
# Add a fully-connected layer of 800 units, using the linear rectifier, and
# initializing weights with Glorot’s scheme (which is the default anyway):
n_hid1 = 4
l_hid1 = lasagne.layers.DenseLayer(
l_in, num_units=n_hid1,
nonlinearity=lasagne.nonlinearities.rectify,
W=init,
b=init
)
n_hid2 = 8
l_hid2 = lasagne.layers.DenseLayer(
l_hid1, num_units=n_hid2,
nonlinearity=lasagne.nonlinearities.rectify,
W=init,
b=init
)
l_out = lasagne.layers.DenseLayer(
l_hid2, num_units=1,
W=init,
b=init
)
prediction = lasagne.layers.get_output(l_out)
# pdb.set_trace()
sigma = pm.HalfNormal(“sigma”, sigma=1)
out = pm.Normal(‘out’,
mu=prediction,
sigma=sigma,
observed=target_var)
return out

and initialize weight with

class GaussWeights(object):
def init(self):
self.count = 0
def call(self, shape):
self.count += 1
return pm.Normal(‘w%d’ % self.count, mu=0, sd=.1,
testval=np.random.normal(size=shape).astype(np.float64),
shape=shape)

and

with pm.Model() as neural_network:
likelihood = build_ann(GaussWeights())

with neural_network:
infrence=pm.ADVI()
mean_field=pm.fit(100000,method=infrence)
trace=mean_field.sample(5000)

2

Does it work without PyMC3? Just Lasagne with optimization.

3

yes. as I mentioned, it works with raw theano (Just Lasagne with optimization). exactly the same hidden layers, neurons and hyperparameters.
my second question: it is true to transfer weight from a raw theano model (suppose results are acceptable) to pymc model (if all parameters and structure are same) as initial weight and then training it.

4

I also had the experience that regression models just don’t work, never figured out why. I think trying to copy over weights is a good thing to try. Please let us know if you figure it out.

1 Like