WAIC values differing between identical fits?

Questions
1

Apologies for any dumb questions, I’m still somewhat new to Bayesian modeling.

I have a fairly complex model. I’m fitting it using ADVI. It appears to give reasonable values for the PPC and posteriors after fitting.

I also have a simpler version of it. I wanted to compare them using WAIC, like they did here. However, I noticed that even with only the complex model (forget about the simpler one), if I fit it repeatedly with the same exact priors/etc, it gives pretty different results for the WAIC.

For example, here it is, running 5 times in a row:

WAIC_r(WAIC=754.5897, WAIC_se=58.94195960, p_WAIC=125.37287431940513, var_warn=1)
WAIC_r(WAIC=764.3613, WAIC_se=58.35469686, p_WAIC=130.48467354412645, var_warn=1)
WAIC_r(WAIC=876.19981, WAIC_se=132.074058, p_WAIC=185.76491713018044, var_warn=1)
WAIC_r(WAIC=785.0630, WAIC_se=63.8712588, p_WAIC=141.14128686850108, var_warn=1)
WAIC_r(WAIC=870.52284, WAIC_se=115.83628, p_WAIC=183.250744816998, var_warn=1)

In addition, I’m worried because they all give this message when doing pm.WAIC:

UserWarning: For one or more samples the posterior variance of the
        log predictive densities exceeds 0.4. This could be indication of
        WAIC starting to fail see http://arxiv.org/abs/1507.04544 for details

So I have a couple questions:

  • Is the variance in these different WAIC values something to worry about, or do we expect about this much difference between (identical) runs just due to random sampling variance?
  • How can I figure out what might be going wrong?
  • Could this be a result of bad convergence or something?

thanks for any advice.

1 Like