Apologies for any dumb questions, I’m still somewhat new to Bayesian modeling.
I have a fairly complex model. I’m fitting it using ADVI. It appears to give reasonable values for the PPC and posteriors after fitting.
I also have a simpler version of it. I wanted to compare them using WAIC, like they did here. However, I noticed that even with only the complex model (forget about the simpler one), if I fit it repeatedly with the same exact priors/etc, it gives pretty different results for the WAIC.
For example, here it is, running 5 times in a row:
WAIC_r(WAIC=754.5897, WAIC_se=58.94195960, p_WAIC=125.37287431940513, var_warn=1) WAIC_r(WAIC=764.3613, WAIC_se=58.35469686, p_WAIC=130.48467354412645, var_warn=1) WAIC_r(WAIC=876.19981, WAIC_se=132.074058, p_WAIC=185.76491713018044, var_warn=1) WAIC_r(WAIC=785.0630, WAIC_se=63.8712588, p_WAIC=141.14128686850108, var_warn=1) WAIC_r(WAIC=870.52284, WAIC_se=115.83628, p_WAIC=183.250744816998, var_warn=1)
In addition, I’m worried because they all give this message when doing
UserWarning: For one or more samples the posterior variance of the log predictive densities exceeds 0.4. This could be indication of WAIC starting to fail see http://arxiv.org/abs/1507.04544 for details
So I have a couple questions:
- Is the variance in these different WAIC values something to worry about, or do we expect about this much difference between (identical) runs just due to random sampling variance?
- How can I figure out what might be going wrong?
- Could this be a result of bad convergence or something?
thanks for any advice.