# Error in Defining Bayesian NN with Multivariate Gaussians

**URL:** <https://discourse.pymc.io/t/error-in-defining-bayesian-nn-with-multivariate-gaussians/13904>\
**Category:** v5\
**Tags:** development, doc, modeling\
**Created:** [February 29, 2024, 9:59am UTC](https://discourse.pymc.io/t/error-in-defining-bayesian-nn-with-multivariate-gaussians/13904 "2024-02-29T09:59:45Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![NourJ](https://avatars.discourse-cdn.com/v4/letter/n/a587f6/32.png) [@NourJ](https://discourse.pymc.io/u/NourJ)\
**Post date:** [February 29, 2024, 9:59am UTC](https://discourse.pymc.io/t/error-in-defining-bayesian-nn-with-multivariate-gaussians/13904/1 "2024-02-29T09:59:45Z")

</div>

Hi there and thanks for all your work,

We are trying to build a Bayesian Neural Network with Multivariate Gaussians as distributions for our parameters. We are following the structure of the code presented in the tutorial

> **[Variational Inference: Bayesian Neural Networks](https://www.pymc.io/projects/examples/en/latest/variational_inference/bayesian_neural_network_advi.html)**
>
> Current trends in Machine Learning: Probabilistic Programming, Deep Learning and “ Big Data ” are among the biggest topics in machine learning. Inside of PP, a lot of innovation is focused on makin...

with univariate Normal assuming a factorization of the posterior.

We counter problems with the incompatibility of dimensions. Here is our code:

```python
def construct_nn(ann_input, ann_output):
    n_hidden = 5 #number of neurons in each hidden layer

    mu_in_1 = np.zeros((X_train.shape[1] * n_hidden), dtype=floatX)
    cov_in_1 = np.eye(X_train.shape[1] * n_hidden, dtype=floatX)

    mu_1_2 = np.zeros((n_hidden * n_hidden), dtype=floatX)
    cov_1_2 = np.eye(n_hidden * n_hidden, dtype=floatX)

    mu_2_out = np.zeros(n_hidden, dtype=floatX)
    cov_2_out = np.eye(n_hidden, dtype=floatX)

    coords = {
        "hidden_layer_1": np.arange(n_hidden),
        "hidden_layer_2": np.arange(n_hidden),
        "train_cols": np.arange(X_train.shape[1]),
        # "obs_id": np.arange(X_train.shape[0]),
    }

    with pm.Model(coords=coords) as neural_network:
        ann_input = pm.Data("ann_input", X_train, mutable=True, dims=("obs_id", "train_cols"))
        ann_output = pm.Data("ann_output", Y_train, mutable=True, dims="obs_id")
        
        # Weights from input to hidden layer 
        #The prior for the weights connecting the input to the first hidden layer is a normal distribution
        weights_in_1 = pm.MvNormal("w_in_1", mu=mu_in_1, cov=cov_in_1, dims=("train_cols", "hidden_layer_1"))
        # Weights from 1st to 2nd layer
        #The prior for the weights connecting the first hidden layer to the second is a normal distribution
        weights_1_2 = pm.MvNormal("w_1_2", mu=mu_1_2, cov=cov_1_2, shape=(n_hidden, n_hidden))

        #The prior for the weights connecting the second hidden layer to the output layer is a normal distribution
        # Weights from hidden layer to output
        weights_2_out = pm.MvNormal("w_2_out", mu=mu_2_out, cov=cov_2_out, dims="hidden_layer_2")

        # Build neural-network using tanh activation function
        act_1 = pm.math.tanh(pm.math.dot(ann_input, weights_in_1))
        act_2 = pm.math.tanh(pm.math.dot(act_1, weights_1_2))
        act_out = pm.math.sigmoid(pm.math.dot(act_2, weights_2_out))

        # Binary classification -> Bernoulli likelihood
        #The likelihood of the model is a Bernoulli distribution, it is used when calling pm.sample() or pm.fit()
        out = pm.Bernoulli(
            "out",
            act_out, #The probability of success for each trial (i.e., the probability of outputting a 1)
            observed=ann_output, #The observed data
            total_size=Y_train.shape[0], # IMPORTANT for minibatches
            dims="obs_id",
        )
    return neural_network

neural_network = construct_nn(X_train, Y_train)

```

The Neural network gets defined but when we train it using this code, we face the following error:

```python
%%time

with neural_network:
    approx = pm.fit(n=30_000) # approx represents the variational approximation to the posterior distribution

```

```python
ValueError: Incompatible Elemwise input shapes [(None, 5), (None, 25)]

```

---

<div class="post-metadata">

**Author:** ![iavicenna](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/iavicenna/32/7923_2.png) [@iavicenna](https://discourse.pymc.io/u/iavicenna)\
**Post date:** [February 29, 2024, 6:06pm UTC](https://discourse.pymc.io/t/error-in-defining-bayesian-nn-with-multivariate-gaussians/13904/2 "2024-02-29T18:06:12Z")

</div>

Your problems arise in here

```auto
 act_1 = pm.math.tanh(pm.math.dot(ann_input, weights_in_1))
 act_2 = pm.math.tanh(pm.math.dot(act_1, weights_1_2))
 act_out = pm.math.sigmoid(pm.math.dot(act_2, weights_2_out))

```

Shape problems are generally very easy to diagnose. Just break inside your code and do evaluations like

```auto
act_1.eval()
act_2.eval()
act_out.eval()

```

You will see that act\_2 and act\_out fails even though all the inputs like weights also .eval() fine. So there is then a problem with the dimensions of things you are trying to dot. act\_1 is shape (500,10) where as weights\_1\_2 is (5,25). Cant dot them