Help Converting Keras Model to PYMC3 syntax

Questions
2

The core of any neural network is just chaining the weights. The L2 regularizer you have is spiritually similar to a normal prior, so that would make a weight layer look like

weight_sd = 0.1 # could use a HalfNormal hyperprior if you really wanted
W1 = pm.Normal('weight_1', 0, weight_sd, shape=(output_dim, input_dim))

Dropout are a set of Bernoulli variables

drop_rate = 0.5
D1 = pm.Bernoulli('dropout_1', drop_rate, shape=(output_dim, input_dim))
WD1 = pm.Deterministic('dropout_weights_1', W1 * D1)

With the weights constructed it’s just multiplication and ReLU

L2 = pm.Deterministic('Layer2', tt.nnet.relu(tt.dot(WD1, L1)))

Bias could be added, if you want

B2 = pm.Normal('bias2', 0., 1.)
L2 = pm.Deterministic('Layer2', tt.nnet.relu(tt.dot(WD1, L1)) + B2)

More generally, see https://twiecki.io/blog/2018/08/13/hierarchical_bayesian_neural_network/

1 Like