I am trying to understand how PyMC3 compute the Bayes Factor in order to check two different instances of a model, with one variable of difference. I am following the docs for the Bayes Factor entry, but here
BF_smc = models[1].marginal_likelihood / models[0].marginal_likelihood
I get ‘AttributeError: ‘Model’ object has no attribute ‘marginal_likelihood’’. I read here that there were instances in which smc gives the wrong answer. Which is the standard way to compare models in PyMC3?
Thank you for the feedback!
The standard approach is called model comparison. You can see an example here
Depending on your versions of PyMC3 and ArviZ the defaults may have changed from that notebook. If scale=deviance in the WAIC or LOO then the lower the number the better, if scale=log then the higher the better. Using the pm.compare() function you can check the weight column of the resulting DataFrame to confirm. In weight the higher the number the better always.
Thank you for the feedback!
I am running pyMC3 version 3.8, so I assume the Bayes Factor entry is deprecated, since the method .marginal_likelihood is not implemented. I will try to use WAIC, which is a new concept for me but according to the reference, is related to Bayes Factor.
I only have partial knowledge of model comparison, but I think Bayes Factors are seen as appropriate for model comparison when one model is a sub-model of another. That is, if one of the parameters is equal to zero then it become equal to a sub-model, then you can just evaluate that one super-model and use BF’s to see if the parameter is credibly zero or not. Eg…
Wagenmakers, E.-J., Lodewyckx, T., Kuriyal, H., & Grasman, R. (2010). Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method. Cognitive Psychology , 60 (3), 158–189. Redirecting