Hello community!
I’m running hierarchical model like this one, except my model has around ~800 priors in total.
I’m getting R^2 scores around ~0.9 and model is predicting with a quite good accuracy on hold-out data. (I also validated hold out data to be sure about R^2)
However, the WAIC scores doesn’t seem to be satisfactory.
WAIC_r(WAIC=-338.8544289866664, WAIC_se=119.20881121134565, p_WAIC=100.20535905846253, var_warn=1)
For one or more samples the posterior variance of the
log predictive densities exceeds 0.4. This could be indication of
WAIC starting to fail see http://arxiv.org/abs/1507.04544 for details.04544 for details
I’ve tried varying hyperparameters and few other variations, but WAIC is more or less same. (Especially it’s in negative!!).
I know WAIC and R^2 are for different aspects and shouldn’t be compared, but should I be concerned about WAIC if I’m getting good R^2?