WAIC and In-Sample Deviance

Thank you, @adam,
I think this shows that you cannot effectively select from or distinguish between polynomial degrees {3,4,5,6}, indicated by dSE > dWAIC. All of them capture the problem; on the other hand, the higher degrees tend to overfit, which is penalized by the WAIC (and even the in-sample deviance). Looking at your data plots and drawing polynomials in there in my mind, this makes intuitive sense.

To share yet a bit of my intuition, the prior should matter only little. In the limiting case of fully data-dominated posterior (infinite number of observations), the model fit is prior-independent, and all conceptually identical models should fit the data equally well. The small difference you observe between the different prior sets are within the SE range. They are not identical because your observations are limited in number.

We should try to understand “why” :wink: dWAIC is just the comfortable display of the difference to the best fitting model, and I suspect dSE is an adjusted standard error that quantifies the uncertainty range of that difference. One could as well use the WAIC and SE: I’m working on a case where my default model is not the WAIC champion, and the better ones are biologically implausible or suffer sampling problems.

Hope this helps!

Falk