Limit or prevent unrealistic output of neural network

Questions
3

Thanks a lot for your reply! I tried

with pm.Model() as neural_network:                        
  weights_in_1 = pm.Uniform('w_in_1', -1, 1,     
                            shape=(X.shape[1], n_hidden1),
                           testval=init_1) 

  weights_1_out = pm.Normal('w_1_out', w_1_out_mu, sd = w_1_out_sd,
                             shape=(n_hidden1,),
                              testval=init_out)

  act_1 = pm.math.tanh( pm.math.dot(ann_input, weights_in_1 ))
  regression = T.dot(act_1, weights_1_out)    #   T.mean(x)>>  mean function in Theano     

   out = pm.Normal('out', mu=regression, sd=np.sqrt( 0.9 ), observed=ann_output)

but got error message

ValueError: Bad initial energy: nan. The model might be misspecified.

Then I tried with [ -10000,10000 ], and it works.
I am quite curious about…is it a good way to figure out how to assign a prior? (Also the bound, I am not sure when I just try and see if it works…) Many papers and articles say the prior is how much information you have for your data before observing them, but in this case I don’t really know…

  1. Is it possible to assign different mu and sd to the nodes in the same hidden layer with the function provided? Because now in the hidden layer (see below), all weights in the first hidden layer all has mu = 0 and sd =1.
  weights_in_1 = pm.Normal('w_in_1', mu = 0, sd = 1,     
                            shape=(X.shape[1], n_hidden1),
                           testval=init_1)

Thanks again for any suggestion.