Hi, I’m trying to train a simple toy model with ADVI. The ELBO seems converged, but the prediction is wrong. The same model works with NUTS sampler, but not with ADVI. My code is here: ADVI PYMC3 · GitHub Thanks! #Toy model def build_toy_dataset(N=50, noise_std=0.2): x = np.linspace(-3, 3,…

The Notebook is private, it’s asking for username and password for access. Could you host it as an open gist instead?

Hi, thanks! I looked into this discussion. it helped a bit but didn’t solve my problem :slight_smile:

Hi, sorry I didn’t notice it is private. Here is the gist: https://gist.github.com/adkoo/99d960455258166b247d3310f439e6db Thanks so much!

I looked into your problem and found a solution. Your model had a very wide prior set for sd. This made it more convenient for the fit to make all weights close to 0, to get the overall data mean correct, and then just increase the standard deviation to make all the observations consistent with nois…

Thanks so much!! I appreciate that. I checked also without the optimizer, and with lower number of steps and it still converges. :slight_smile: Could you please explain why did you delete the ‘s.set_value(0)’ in the inference? and a follow-up question please - how should I choose the prior set in…

[image] Adi_Hanuka: Could you please explain why did you delete the ‘s.set_value(0)’ in the inference? Sorry for the confusion. Before touching the model, I did a bunch of small tests with the optimizer that didn’t work. Removing that line was one of them. I imagine the new model will work wi…

Thanks so much! Yes, I meant this: [image] lucianopaz: About the values, if you have no idea of a potential range, you should try a couple of values that are orders of magnitude apart. That way you check your models sensitivity to the prior hyperparameters. Such a simple toy model is sensit…

[image] Adi_Hanuka: What other build-in function pymc3 has in order to check and verify that? for example, I saw “pm.traceplot” is one option. Do you have other recommendations? I think that @aloctavodia or @colcarroll can give you much better advice than me on this subject. [image] Adi_H…

My suggestion is to make weight init to small values, like normal with 1e-2 std and small initial posterior sigma. Increased hidden dim should also help. The drawback with normal prior is to irreducible over parametrization, you May also try some sparsifying priors like studentT.

BNN regression using ADVI

Questions

junpenglao May 10, 2019, 1:29pm 3

You might also find this previous discussion helpful: Poor Accuracy of ADVI for Linear Regression

Topic		Replies	Views
Poor Accuracy of ADVI for Linear Regression Questions	12	3639	April 18, 2018
Bayesian Neural Network unable to converge on simple model Questions	19	2180	March 4, 2022
Bayesian Neural Network - No convergence Questions	7	1534	July 8, 2019
Average loss for ADVI never decrease for my model General	6	540	September 3, 2023
Using ADVI's question Questions	5	634	December 3, 2018

BNN regression using ADVI

Related topics