Model Comparison, DiscreteUniform vs. Bernoulli, ElemwiseCategorical vs. CategoricalGibbsMetropolis

[junpenglao]

@cs224

LOO is Leave One Out Cross Validation and WAIC is Watanabe–Akaike Information Criterion, correct?

Yep

@cs224

Then you’re basically saying that model bayesian model comparison via a hierarchical model is “out”? Is this because of its technical difficulties (like my problem) or some fundamental theoretical reason?

Mostly because of the problem in inference, you need to write a huge mixture model, so you have

1. difficulty of inferencing a mixture model
2. difficulty of scaling (you have a much larger model now)
3. not flexible, if you want to add a new model you need to rewrite + run all sampling again, whereas LOO or WAIC you just train a new model.

Having said that, there are ways to do it properly, eg see Motif of the Mind | Junpeng Lao, PhD, which based on Christian Robert’s paper: https://arxiv.org/pdf/1412.2044v2.pdf