I just encountered an error when using az.compare to, well, compare two models fit using ADVI; however, I get the following error:
TypeError: Encountered error in ic computation of compare.
I used the samples generated from the fit as traces. Should I be using another object, or is it not possible to compare models fit with ADVI?
Thank you in advance!
I found the same issue recently (pymc 4.x). I find out that the computation of the log likelihood is disable on purpose and thus not part of the InferenceData (@ferrine probably knows the reason).
You can still ask pymc to compute the log likelihood for you by doing
trace = mean_field.sample(1000, return_inferencedata=False)
idata = pm.to_inference_data(trace=trace, model=model, log_likelihood=True)
Then you can use that idata to compute LOO.
While this will get the values computed I will be careful about the results until getting more info from @ferrine or others knowing why it is currently disable by default.
Thank you very much for the reply. I’ll check it out and see if my results make sense, although I will not make any decision until we get more information.
I think the comparison is not fair. Consider the case when your approximate posterior converged to have narrow high density region, while the true posterior has a broader shape. In the comparison, the narrow model would always win which is not true. I’m not against of returning the log likelihood, but this leads to confusion in interpretation
@ferrine If I understand it correctly this paper argues for using PSIS loo cross validation for identifying problems with ADVI approximations. They cite the R loo package, but I believe this is the same as the arviz.loo, which in turn uses the
log_likelihood component from the