I have two questions. The first one is related to the evaluation of models that are not predictive. I am interested on fitting a GLM (Poisson) model to describe the coefficient of the covariates that are correlated with my dependent variable. Thus, I am not interested in out-of-sample predictions.
I’ve seen many papers using DIC as an evaluation metric (e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4282098/). In the pymc3 github I also read about many people suggesting WAIC as an evaluation metric. After reading this paper (“Understanding predictive information criteria for Bayesian models” https://link.springer.com/article/10.1007/s11222-013-9416-2) my questions are:
Isn’t WAIC “just” related to out-of-sample/generalization prediction? So does it make sense to use it in my project?
What should I do when I have these warnings? Can I solve them somehow? (not clear to me how). If not, can I trust WAIC?
DIC 217.868591309 /home/nadai/.local/lib/python3.4/site-packages/pymc3/stats.py:213: UserWarning: For one or more samples the posterior variance of the log predictive densities exceeds 0.4. This could be indication of WAIC starting to fail see http://arxiv.org/abs/1507.04544 for details """) WAIC WAIC_r(WAIC=158.42521828487168, WAIC_se=2.3092993247226805, p_WAIC=7.2801294) /home/nadai/.local/lib/python3.4/site-packages/pymc3/stats.py:278: UserWarning: Estimated shape parameter of Pareto distribution is greater than 0.7 for one or more samples. You should consider using a more robust model, this is because importance sampling is less likely to work well if the marginal posterior and LOO posterior are very different. This is more likely to happen with a non-robust model and highly influential observations. happen with a non-robust model and highly influential observations.""") LOO LOO_r(LOO=164.91826, LOO_se=2.4069428499178245, p_LOO=10.526650309191908)
Are papers using DIC in hierarchical models all wrong?