Basic usage problem within PyMC3

Hello everyone! I am a beginneer to PyMC3, and I got a problem when I try to implement Bayesian Neural Networks (BNN) with PyMC3.

The problem is that I want to combine GMM with BNN during the training process of BNN with NUTS. When I run the following simple code, the problem raise.

####### The problrm code #######

import numpy
from sklearn import mixture
import theano
import theano.tensor as T
import pymc3 as pm

input_nodes = 50 #Dimension of the objective problem
hidden_nodes1 = 10
hidden_nodes2 = 10
hidden_nodes3 = 50

loc1=1
scale1=0.2

#Normal Sampling
x_train = scale1*numpy.random.randn(1000,50)+loc1

y_train = 2*x_train+1

def construct_NN(x_train, y_train):

# set number of nodes in each input,hidden,output layer
# set number of nodes in each input,hidden,output layer
inodes  = input_nodes #input layer
hnodes1 = hidden_nodes1
hnodes2 = hidden_nodes2
hnodes3  = hidden_nodes3

#Initial parameter for NN
wh1 = numpy.random.normal(0.0, pow(hnodes1,-0.5), (inodes, hnodes1))

b1  = numpy.random.normal(0.0, pow(hnodes1,-0.5), (hnodes1,))

wh2 = numpy.random.normal(0.0, pow(hnodes2,-0.5), (hnodes1, hnodes2))

b2  = numpy.random.normal(0.0, pow(hnodes2,-0.5), (hnodes2,))

wh3 = numpy.random.normal(0.0, pow(hnodes2,-0.5), (hnodes2, hnodes3))

b3  = numpy.random.normal(0.0, pow(hnodes2,-0.5), (hnodes3,))

with pm.Model() as neural_network:
    
    ann_input = x_train #pm.Data('ann_input1', x_train)
    ann_output = y_train #pm.Data('ann_output1', y_train)
    
    # Weights from input to hidden layer
    weights_in_1 = pm.Normal('w_in_1', 0, sigma=1,
                             shape=(input_nodes, hidden_nodes1),
                             testval=wh1)
    
    # Bias from input to hidden layer
    weights_in_b = pm.Normal('w_in_b1', 0, sigma=1,
                             shape=(hidden_nodes1,),
                             testval=b1)        
    
    # Weights from 1st to 2nd layer
    weights_in_2 = pm.Normal('w_in_2', 0, sigma=1,
                            shape=(hidden_nodes1, hidden_nodes2),
                            testval=wh2)
    
    # Bias from input to hidden layer
    weights_in_b2 = pm.Normal('w_in_b2', 0, sigma=1,
                             shape=(hidden_nodes2,),
                             testval=b2)     
    
    # Weights from 2nd to 3rd layer
    weights_in_3 = pm.Normal('w_in_3', 0, sigma=1,
                            shape=(hidden_nodes2, hidden_nodes3),
                            testval=wh3)
    
    # Bias from 2nd to 3rd layer
    weights_in_b3 = pm.Normal('w_in_b3', 0, sigma=1,
                             shape=(hidden_nodes3,),
                             testval=b3)     
    
    ϵ = pm.HalfCauchy('ϵ', 5)
    
    ######################################################
    #Fit the Multivariate distribution 
    ######################################################
    
    hidden_outputs1 = pm.math.tanh(pm.math.dot(ann_input,weights_in_1)+weights_in_b)
    
    hidden_outputs2 = pm.math.tanh(pm.math.dot(hidden_outputs1,weights_in_2)+weights_in_b2)  

    hidden_outputs3 = pm.math.tanh(pm.math.dot(hidden_outputs2,weights_in_3)+weights_in_b3) 
    
    #Bayesian Gaussian Mixture Model
    PDF = mixture.BayesianGaussianMixture(n_components=20,n_init=5,
                                          covariance_type='full').fit(hidden_outputs3)
    
    μ = sum(PDF.means_)

    y_pred = pm.Normal('y_pred', mu=μ, sd=ϵ, observed=ann_output)

return neural_network

neural_network = construct_NN(x_train, y_train)

with neural_network:

trace_pce = pm.sample(5000,chains=4)

####### The error output here #######

When I run this code I got the following error information :
“ValueError: setting an array element with a sequence.”

I guss this error comes from :
“PDF = mixture.BayesianGaussianMixture(n_components=20,n_init=5,
covariance_type=‘full’).fit(hidden_outputs3)”

Because the at this stage the term “hidden_outputs3” is not an array. And I don’t known how to deal with this problem.

####### Versions and main components #######

  • PyMC3 Version: 3.8
  • Theano Version: 1.0.4
  • Python Version: 3
  • Operating system: Windows 10
  • How did you install PyMC3: (conda/pip): pip install

Could you please give me some advices? Thanks for your help.

The shape of hidden_outputs3 is 1000 x 50. All the hidden outputs are arrays, I think because hidden_outputs1 depends on weights_in_1, and that carries through for all of them

Thanks! I think the problem stems from the following aspect:" during the sampling process of NUTS the newly accepted samples of parameters are in closed form, which can not be passed back to the network, and the code need to recompute GMM every time the newly accepted samples of parameters are founded. What I need is something like BP algorithm, the updation can be obtained with the iteration"

Hi Jingfei

I don’t think you can mix sklearn BayesianGaussianMixture with PyMC3. The SciPy library usually calculates the single point which maximises the likelihood whereas PyMC (and other bayesian PPL languages) give a full posterior distribution.

Maybe if you could describe what you want to achieve then we can see if PyMC can help.

Yes! A lot of Thanks! PyMC gives a full posterior distribution of the parameters, and the posterior distribution computed by NUTS is actually composed of the accepted samples of HMC. In such case, I need the accepted samples of HMC at each iteration, because GMM is also changed with the procedure of sampling process of NUTS. However, I don’t know how to query the accepted samples of HMC at each iteration, the trace operation within PyMC can only give all of the accepted samples after the inference process of NUTS is finished.

Just trying to understand this problem in order to better understand how pymc3 works - I gather I misunderstood the original question, my apologies. I had thought that hidden_outputs3 would pass a 1000x50 array to PDF each time a sample is drawn, mostly based on my impression of how values are passed between pymc3 nodes. Is that not the case?

Yes, what I need is the accepted samples of NUTS at each iteration, because the target model is also changed after each sample is drawn, maybe you can help to run the code, so that you can give better help, thank you very much!