I don’t have a lot of experience working with Bayesian neural nets, but I’ve read trying to do full MCMC on them is very difficult, so ADVI is the way to go. There’s an example of a bayesian neural network using ADVI in the examples gallery here. The exact same model as the example can be sampled however you wish, though.
The example uses tanh activations, but aesara has the full suite of neural network stuff in aesara.tensor, for example aesara.tensor.nnet.relu and aesara.tensor.sigmoid (or pm.invlogit, which is equivalent). Check the aesara docs for more info.