Poor Accuracy of ADVI for Linear Regression

Questions
3

Thanks for the suggestions @junpenglao

  1. Yes, I check the convergence. I have plotted the ELBO history. It seems that ADVI has already converged when passing 6000th iteration. Since our model is a simple linear regression model, I expected that to converge within 10000 iterations. Moreover, I tried out with several other datasets and observed poor accuracy for ADVI. To be sure, I also tried increasing the number of iterations to 50000 and the model still gives similar error.
    1

  2. After you recommended, I tried using ADAM and still I observe an error close to SGD.

  3. I tried increasing the number of MC samples, and yet ADVI gives the same accuracy.

It seems that we are missing something here. This task is not that difficult (this is not some hierarchical model or BDL) for both ADVI and MCMC, and therefore they both should behave as expected even with the default configurations.

I would love to look at your notebook, can you please provide me with the link to that.