Comparing Models with WAIC

Maybe a little late, but I want to recommend you to read chapter 6 of Statistical Rethinking, in that Chapter Richard McElreath discuss the question How big a difference in WAIC is “significant”? Spoiler alert he says “In general, it is not possible to provide a principled threshold of difference that makes one model “significantly” better than another, whatever that means”.

You can use the function pm.compare() and pm.compareplot to present the result of your analysis and help analyze it. Both these function are based on the mentioned chapter. If you use pm.compare() with the argument method=BB-pseudo-BMA the standard errors of WAIC are computed using Bayesian Bootstrapping, this is a good idea given that WAIC distribution could be skewed. Also notice that pWAIC is and estimation of the effective parameters in your models, probably you should not take that number too seriously but I guess it can give you and idea of the flexibility/complexity of your model, that may help you discuss your results ans answer the reviewer. Notice that hierarchical models are less complex than they seems, this is related to the shrinkage effect.

2 Likes