# Deep Neural Network Question

**URL:** <https://discourse.pymc.io/t/deep-neural-network-question/2793>\
**Category:** Questions\
**Created:** [February 24, 2019, 7:39pm UTC](https://discourse.pymc.io/t/deep-neural-network-question/2793 "2019-02-24T19:39:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jordan.howell2](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jordan.howell2/32/1742_2.png) [@jordan.howell2](https://discourse.pymc.io/u/jordan.howell2)\
**Post date:** [February 24, 2019, 7:39pm UTC](https://discourse.pymc.io/t/deep-neural-network-question/2793/1 "2019-02-24T19:39:17Z")

</div>

So I’ve gotten a few Bayesian neural networks to run. I haven’t been able to wrap my head around understanding how to make a layer wider. For example, a keras/tensor flow model may have a layer with 256 nodes, and another with 128, 64, etc.

How does that compare, in terms of coding using pymc3? Or does that matter when using Bayesian methods?

---

<div class="post-metadata">

**Author:** ![chartl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/chartl/32/1515_2.png) [@chartl](https://discourse.pymc.io/u/chartl)\
**Post date:** [February 24, 2019, 8:41pm UTC](https://discourse.pymc.io/t/deep-neural-network-question/2793/2 "2019-02-24T20:41:13Z")

</div>

Just change the size of the weights (W) for each layer, such as:

```auto
with pm.Model() as two_layer:
  W1 = pm.Normal('l1_weight', 0., 1e-3., shape=(256, input_size))
  W2 = pm.Normal('l2_weight', 0., 1e-3, shape=(128, 256))
  L1 = pm.Deterministic('Layer1', tt.nnet.relu(tt.dot(W1, input_layer)))
  L2 = pm.Deterministic('Lyaer2', tt.nnet.relu(tt.dot(W2, L1)))
  ...

```

Edit: originally I mis-specified the keyword `shape=` as `size=`

---

<div class="post-metadata">

**Author:** ![jordan.howell2](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jordan.howell2/32/1742_2.png) [@jordan.howell2](https://discourse.pymc.io/u/jordan.howell2)\
**Post date:** [February 24, 2019, 11:13pm UTC](https://discourse.pymc.io/t/deep-neural-network-question/2793/3 "2019-02-24T23:13:04Z")

</div>

Once again @chartl to my rescue! Thank you for the help. Could I bother you with one more question? What is the difference between

> trace = pm.sample(draws=15000, init=‘advi’, progressbar=True)

and

> with neual\_network:  
> inference = pm.ADVI()  
> approx = pm.fit(n=50000, method=inference)  
> trace = approx.sample(draws=5000)

What is actually happening in the background with the two?

---

<div class="post-metadata">

**Author:** ![jordan.howell2](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jordan.howell2/32/1742_2.png) [@jordan.howell2](https://discourse.pymc.io/u/jordan.howell2)\
**Post date:** [February 25, 2019, 11:07am UTC](https://discourse.pymc.io/t/deep-neural-network-question/2793/4 "2019-02-25T11:07:06Z")

</div>

Well I thought this worked. Now it’s saying this:

> TypeError: **init** () got an unexpected keyword argument ‘size’

I’m trying this on the first weight.

> n\_hidden = 10
> 
> # Initialize random weights between each layer
> 
> init\_1 = np.random.randn(X\_train.shape[1], n\_hidden)  
> init\_2 = np.random.randn(n\_hidden, n\_hidden)  
> init\_3 = np.random.randn(n\_hidden, n\_hidden)  
> init\_4 = np.random.randn(n\_hidden, n\_hidden)  
> init\_5 = np.random.randn(n\_hidden, n\_hidden)  
> init\_6 = np.random.randn(n\_hidden, n\_hidden)  
> init\_7 = np.random.randn(n\_hidden, n\_hidden)  
> init\_8 = np.random.randn(n\_hidden, n\_hidden)  
> init\_9 = np.random.randn(n\_hidden, n\_hidden)  
> init\_10 = np.random.randn(n\_hidden, n\_hidden)  
> init\_out = np.random.randn(n\_hidden)
> 
> with pm.Model() as neual\_network:  
> # Weights from input to hidden layer  
> weights\_in\_1 = pm.Normal(‘w\_in\_1’, 0, sd=1,  
> shape = (X\_train.shape[1], n\_hidden),  
> size = (128, X\_train.shape[1]),  
> testval=init\_1)
> 
> ```
> # Weights from 1st to 2nd layer
> weights_1_2 = pm.Normal('w_1_2', 0, sd=1, 
> shape = (128, n_hidden), 
> testval=init_2)
> 
> # Weights from 1st to 2nd layer
> weights_2_3 = pm.Normal('w_2_3', 0, sd=1, 
> shape=(64, 128), 
> testval=init_3)
> 
> # Weights from 1st to 2nd layer
> weights_3_4 = pm.Normal('w_3_4', 0, sd=1, 
> shape=(64, 64), 
> testval=init_4)
> 
> # Weights from 1st to 2nd layer
> weights_4_5 = pm.Normal('w_4_5', 0, sd=1, 
> shape=(32, 64), 
> testval=init_5)
> 
> # Weights from 1st to 2nd layer
> weights_5_6 = pm.Normal('w_5_6', 0, sd=1, 
> shape=(32, 32), 
> testval=init_6)
> 
> # Weights from 1st to 2nd layer
> weights_6_7 = pm.Normal('w_6_7', 0, sd=1, 
> shape=(32, 16), 
> testval=init_7)
> 
> # Weights from 1st to 2nd layer
> weights_7_8 = pm.Normal('w_7_8', 0, sd=1, 
> shape=(16, 16), 
> testval=init_8)
> 
> # Weights from 1st to 2nd layer
> weights_8_9 = pm.Normal('w_8_9', 0, sd=1, 
> shape=(8, 16), 
> testval=init_9)
> 
> # Weights from 1st to 2nd layer
> weights_9_10 = pm.Normal('w_9_10', 0, sd=1, 
> shape=(1, ), 
> testval=init_10)
> 
> # Weights from hidden layer to output
> #weights_10_out = pm.Normal('w_10_out', 0, sd=1, 
> # shape=(n_hidden,), 
> # testval=init_out)
> 
> # Build neural-network using relu activation function
> B2 = pm.Normal('bias2', 0., 1.)
> 
> act_1 = T.nnet.relu(T.dot(ann_input, weights_in_1))
> 
> act_2 = T.nnet.relu(T.dot(act_1, weights_1_2))
> 
> act_3 = T.nnet.relu(T.dot(act_2, weights_2_3) + B2)
> 
> act_4 = T.nnet.relu(T.dot(act_3, weights_3_4))
> 
> act_5 = T.nnet.relu(T.dot(act_4, weights_4_5) + B2)
> 
> act_6 = T.nnet.relu(T.dot(act_5, weights_5_6))
> 
> act_7 = T.nnet.relu(T.dot(act_6, weights_6_7) + B2)
> 
> act_8 = T.nnet.relu(T.dot(act_7, weights_7_8))
> 
> #act_9 = T.nnet.relu(T.dot(act_8, weights_9_10) + B2) 
> 
> act_out = T.dot(act_8, weights_9_10)
> 
> out = pm.Normal('out', mu = act_out, observed=ann_output, shape = y_train_t.shape)
> 
> ```

---

<div class="post-metadata">

**Author:** ![chartl](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/chartl/32/1515_2.png) [@chartl](https://discourse.pymc.io/u/chartl)\
**Post date:** [February 25, 2019, 5:00pm UTC](https://discourse.pymc.io/t/deep-neural-network-question/2793/5 "2019-02-25T17:00:00Z")

</div>

> [@jordan.howell2](#):
>
> Once again @chartl to my rescue! Thank you for the help. Could I bother you with one more question? What is the difference between
> 
> > trace = pm.sample(draws=15000, init=‘advi’, progressbar=True)
> 
> and
> 
> > with neual\_network:  
> > inference = pm.ADVI()  
> > approx = pm.fit(n=50000, method=inference)  
> > trace = approx.sample(draws=5000)
> 
> What is actually happening in the background with the two?

The former runs the NUTS sampler, but with an initial matrix initialized by ADVI. The latter just runs ADVI.

> [@jordan.howell2](#):
>
> Well I thought this worked. Now it’s saying this:
> 
> > TypeError: **init** () got an unexpected keyword argument ‘size’

That might actually be in my example, it looks like you’ve got a `size=` somewhere there should be a `shape=`. In fact `weights_in_1` has **both** a `size=` and a `shape=`.