I am comparing a set of mixed models, sampled with PyMC 4.0 with a few predictors and slight changes in the random slopes/intercept structures between the models. I compare the output with az.compare (with both loo and waic). My first ranked model has the highest weight, but my 3ranked model has higher weight than the 2nd… is this possible? In fact, I ran the models on three different datasets, and I always get the same best model, lower ranked models all show such inconsistencies. I tried both loo and waic, they give essentially the same result, with no warnings for loo. any ideas if this possible or something is really off?
1 Like
Yes, this can happen and it is not that uncommon. See Bayesian model averaging: ranking of model weights and LOO don't match for more detailed answer
2 Likes
Thanks!